Mojibuster

Fix character encoding in PowerShell

Your script writes é or ? instead of é, or the console shows garbled text? Windows PowerShell 5.1 uses a different encoding depending on the command. Here are the causes and the right parameters – and you can repair broken files in your browser above.

Works with text and these files:

  • CSV
  • TXT
  • JSON
  • XML
  • SRT
  • VTT
  • TSV
  • MD
  • HTML
  • SQL
  • LOG

For example Excel exports, database dumps, subtitles or website code.

Repair

  1. Drop a file or paste text
  2. It gets repaired automatically
  3. Copy or download
or simply drag a file here

The root problem: every command, a different encoding

In Windows PowerShell 5.1, almost every command has its own default encoding: Get-Content reads files without a BOM as ANSI (Windows-1252), Out-File and > write UTF-16, Set-Content writes ANSI and Export-Csv ASCII. PowerShell 7 unified this and uses UTF-8 without a BOM everywhere.

A UTF-8 file read with Get-Content in PowerShell 5.1

Before

User;Department;Office
José García;Sales;Málaga
Renée Dubois;Purchasing;Zürich

After

User;Department;Office
José García;Sales;Málaga
Renée Dubois;Purchasing;Zürich

Read and write files correctly

  • Get-Content data.csv -Encoding UTF8 reads UTF-8 files correctly even without a BOM.
  • Set-Content, Out-File and Add-Content write UTF-8 with -Encoding UTF8 – in 5.1 always with a BOM.
  • Export-Csv data.csv -Encoding UTF8 -NoTypeInformation instead of the ASCII default, which replaces accented letters with ?.
  • For all commands at once: $PSDefaultParameterValues['*:Encoding'] = 'utf8' at the top of the script.

Web requests with Invoke-RestMethod and Invoke-WebRequest

If a server doesn’t state a character set in its Content-Type, Windows PowerShell 5.1 decodes the response as ISO-8859-1. UTF-8 responses then contain é. The fix: fetch the response as bytes and decode it yourself, for example with [Text.Encoding]::UTF8.GetString((Invoke-WebRequest $url).RawContentStream.ToArray()).

Script files with special characters

If you save a script as UTF-8 without a BOM, Windows PowerShell 5.1 reads it as ANSI. Characters in strings like Write-Host "Café" then show up as Café. Save .ps1 files as UTF-8 with a BOM – in VS Code via the encoding in the status bar, in Notepad++ via Encoding → UTF-8-BOM.

The console

If only the console window shows wrong characters, the file is often fine. Switch the output with [Console]::OutputEncoding = [Text.Encoding]::UTF8. For output you pipe to external programs, also set $OutputEncoding = [Text.Encoding]::UTF8.

If the file has already been written broken, drop it into the box above: Mojibuster repairs ̩ and friends and converts ANSI files to UTF-8. If the file already contains question marks, the text is lost Рsee question marks instead of characters.

Frequently asked questions

Which PowerShell version do I have?

$PSVersionTable.PSVersion shows it. 5.1 is the Windows PowerShell built into Windows; from 7 on it’s just called PowerShell and uses UTF-8 by default.

Why does Excel open my CSV from PowerShell incorrectly?

PowerShell 7 writes UTF-8 without a BOM, which Excel doesn’t recognize. Use -Encoding utf8BOM or save the file with a BOM in Mojibuster. More on Excel showing special characters wrong.

Do I need chcp 65001?

Only for programs that follow the console code page. Within PowerShell, [Console]::OutputEncoding is enough.

More guides