The Windows Github Desktop program is an easy to use Git GUI for Windows users and it works great in most situations. However, it has a notorious history of converting every file in a Git repository checkout/clone to all CRLF (Windows Style) line endings. Line endings are also known as EOL's (End of Line).
To make line endings visible in Notepad++ go to
View > Show Symbols > Show End of Line
Here's an example of what CRLF's look like when made visible.
There are ways of manually converting EOL's to LF using Notepad++
Edit > EOL Conversion > Unix (LF)
You can also setup Macro's in Notepad++ to automate this process by running a macro on each file but it will only do 1 file at a time and you must manually save each file. Converting a large amount of bulk files cannot be automated easily.
They're just invisible line endings! Who cares!?!?
The Windows Githhub Desktop app becomes a huge problem if you intend to contribute to a large open source project like Adafruit's Circuit Python. Pre-commit is a normal part of the process and runs Pylint. Pylint only accepts LF (Unix Style) EOL's and will throw thousands of CRLF line ending errors. So yes it matters, a lot.
As you can see in the below image, any legitimate errors that Pylint raises during pre-commit gets buried in hundreds of errors for
There is CRLF while it should be LF
All due to Github Desktop creating them!
All Github Desktop had to do is add a preference option to checkout and commit with LF only. No such option exists. This behavior is not present in any other Git based tool so this is a problem unique for Windows Github Desktop users. There are workarounds published online that include changing an environment variable and global git config. That workaround only works for 1 session, as soon as you close Github Desktop it will promptly ignore your previous changes. There are hundreds of issues filed about this in the Github Desktop repo over the past decade with no real solution.
There has to be a way to fix this!
Often times what cannot be done with a Windows batch script can be done with a Windows PowerShell script. Think of PowerShell as a far more capable version of Command Prompt. To be fully transparent I used ChatGPT to do most of the work and here is the ChatGPT Session. Make robot fren not robot enemy.
Here is the powershell script called AutoLF.ps1 (ps1 is a powershell script extension, it has nothing to do with PlayStation).
There are 2 ways to use this script.
- Drag and drop it into a directory (making sure to uncomment $folderPath)
- Specify a directory path to run it on.
Here I'm using Option 2 and have uncommented both $username and $folderPath lines by removing the # at the beginning of the line.
# SPDX-FileCopyrightText: 2024 DJDevon3
#
# SPDX-License-Identifier: MIT
# Attribution: ChatGPT https://chat.openai.com/share/4fe18cf7-4199-431f-b4de-60a8e5a2c232
# ------------- Choose directory pathing option ----------------
# Script will run on the directory you choose and all subdirectories within.
# Option 1: Run in the same directory as this file (drag/drop/run)
#$folderPath = $PSScriptRoot
# Option 2: Set the folder path manually
$username = $env:USERNAME
$folderPath = "C:\Users\$username\Documents\GitHub\Adafruit_CircuitPython_Requests\"
# Change to the desired directory. Uses Option 1 or 2 automatically as both use the same variable folderPath.
Set-Location -Path $folderPath
# ----------------- Gather environment info ----------------------
$currentDate = Get-Date -Format "yyyy-MM-dd"
$psVersion = $PSVersionTable.PSVersion
# ----------------- Begin Script Output ---------------------------
# echo/print version info
Write-Host "Date: $currentDate"
Write-Host "PowerShell version: $($psVersion.Major).$($psVersion.Minor)"
Write-Host ""
# Get a list of .py files in the repo directory and included subdirectories
$pyFiles = Get-ChildItem -Path $folderPath -Filter *.py -Recurse
Write-Host "------------------------ CONVERTING LINE ENDINGS (EOL) TO LF ------------------------"
Write-Host ""
# Powershell will loop through each .py file converting CRLF to LF.
foreach ($file in $pyFiles) {
# Setup ignored .py files here. 'conf.py' for example. We'e only cleaning up .py files.
if ($file.Name -eq 'conf.py') {
Write-Host "Ignored: $($file.FullName): File named 'conf.py' is ignored."
Write-Host ""
continue
}
# Read the content of the file
$content = Get-Content -Path $file.FullName -Raw
# Replace \r\n to \n line endings
$content = $content -replace "`r`n", "`n"
# Write the modified content back to the file
Set-Content -Path $file.FullName -Value $content
Write-Host "EOL's converted to LF for $($file.FullName)"
Write-Host ""
}
# Keep the PowerShell window open after completing to inspect for any errors.
Write-Host "Press any key to exit..."
$null = $Host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
Save the contents to a file named AutoLF.ps1 You can rename it whatever you want, keeping the .ps1 file extension. To execute the script simply right-click on it and choose "Run with PowerShell".
If you're used to Command Prompt being a big black terminal then say hello to the very blue PowerShell terminal. Here you can see the script executed on the file path I specified and converted every .py (python) file from CRLF line endings to LF line endings. Very quick and easy!
and the result of automated LF conversion using this method.
The Github Desktop CRLF problem now has a 1-click solution!
While the simple version does fix CRLF errors there's a lot more we could do with it. We can include pre-commit install, black, reuse, and pre-commit run -all-files to run with PowerShell!
Here is a script I've named AutoLF_Pre-Commit.ps1 It has the same configuration as the AutoLF.ps1 but also converts other things and runs pre-commit automatically. Now you'll know your repo has cleaned up the issues introduced by Github Desktop before working on a newly cloned repo.
All you have to do is remember to run it immediately after cloning a new repository and integrate into your normal workflow.
# SPDX-FileCopyrightText: 2024 DJDevon3
#
# SPDX-License-Identifier: MIT
# Attribution: ChatGPT https://chat.openai.com/share/4fe18cf7-4199-431f-b4de-60a8e5a2c232
# ------------- Choose directory pathing option ----------------
# Option 1: Run in the same directory as this file (drag/drop/run)
#$folderPath = $PSScriptRoot
# Option 2: Set the folder path manually
# folderPath should be the top level repository directory, not a subdirectory.
# good example C:\Users\$username\Documents\GitHub\Adafruit_CircuitPython_Requests\
# bad example C:\Users\$username\Documents\GitHub\Adafruit_CircuitPython_Requests\wifi\api\expanded
$username = $env:USERNAME
$folderPath = "C:\Users\$username\Documents\GitHub\Adafruit_CircuitPython_Requests\"
# Change to the desired directory. Uses Option 1 or 2 automatically as both use the same variable folderPath.
Set-Location -Path $folderPath
# ----------------- Gather environment info ----------------------
$currentDate = Get-Date -Format "yyyy-MM-dd"
$pythonVersion = & python --version 2>&1
$preCommitVersion = & pre-commit --version 2>&1
$blackVersion = & black --version 2>&1
$reuseVersion = & reuse --version 2>&1
# ----------------- Begin Script Output ---------------------------
# echo/print version info
Write-Host "Date: $currentDate"
Write-Host "Python: $pythonVersion"
Write-Host "pre-commit: $preCommitVersion"
Write-Host "black: $blackVersion"
Write-Host "reuse: $reuseVersion"
Write-Host ""
Write-Host "----- Attempting pre-commit install on ($folderPath) -----"
# This will do nothing if it's previously been run.
try {
& 'pre-commit' install --install-hooks --overwrite
} catch {
Write-Host "Error installing pre-commit hooks in the directory: $_"
}
Write-Host ""
Write-Host "----- PIP INSTALL --UPGRADE REUSE -----"
# This will do nothing if it's already up-to-date.
try {
& pip install --upgrade reuse
} catch {
Write-Host "Error upgrading reuse tool: $_"
}
Write-Host ""
# Get a list of .py files in the repo directory and included subdirectories
$pyFiles = Get-ChildItem -Path $folderPath -Filter *.py -Recurse
Write-Host "------ Convert line endings to LF then format with black ------"
Write-Host ""
# Powershell will loop through each .py file converting CRLF to LF.
foreach ($file in $pyFiles) {
# Setup ignored .py files here. 'conf.py' for example. We'e only cleaning up .py files.
if ($file.Name -eq 'conf.py') {
Write-Host "Skipping file $($file.FullName): File named 'conf.py' is ignored."
continue
}
# Read the content of the file
$content = Get-Content -Path $file.FullName -Raw
# Replace \r\n to \n line endings
$content = $content -replace "`r`n", "`n"
# Write the modified content back to the file
Set-Content -Path $file.FullName -Value $content
Write-Host "EOL's converted to LF for $($file.FullName)"
# Run black code formatter on the file
try {
& 'black' $file.FullName
} catch {
Write-Host "Error running black for $($file.FullName): $_"
}
}
# Add a blank line
Write-Host ""
Write-Host "------------- PRE-COMMIT RUN --ALL-FILES -------------"
# Since all py files have already been changed to LF and formatted
# there shouldn't be hundreds of line-ending errors on pre-commit.
# Pre-commit runs on all files in the repo not just py files
try {
& 'pre-commit' run --all-files
} catch {
Write-Host "Error running pre-commit for files in $($folderPath): $_"
}
Write-Host "-------------------------------------------------------------------------"
# Pause at the end to keep the PowerShell window open
Write-Host "Press any key to exit..."
$null = $Host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
The result is a repo with .py files completely cleaned of Github Desktop CRLF files then run with black and pre-commit.
No more hundreds (or thousands) of CRLF errors when running pre-commit. This is the very satisfying result.
If you're worried about all these line endings changing files that will then be added to your commit don't be!
Click on Commit and you'll find that Github Desktop will act like it will submit the changes at first. This is a hair raising experience (if it's not your personal repo) that you're going to add a bunch of changes you didn't want to! Not to worry, they won't actually commit as you'll see.
As long as your commits are white space or EOL changes only they will not actually commit.
It will not throw an error but the message in the center of the screen will change to No content changes found
You can optionally choose the gear icon in the top right (only appears during a commit) and select
Hide whitespace changes
Then it will recognize there are no file changes other than whitespace or EOL changes and throw an error
There are no changes to commit
Github Desktop will then automatically remove all the files that were just in your commit because they're EOL changes only. They vanish. Then Github Desktop reloads you back to your repo main screen (home) and all is right with the world again.
This is a typical commit workflow for a Github Desktop user. Now that you're armed with the AutoLF powershell script you can remove all the obstacles that have prevented you from making easy pre-commits and keeping your workflow efficient and productive.
Github Repository: DJDevon3's WindowsPowerShellScripts