proposal: debug/pe: add BigObj COFF support via new fields on File struct

[Peter0x44]

Proposal Details

The debug/pe package currently uses this file struct:

// A File represents an open PE file.
type File struct {
	FileHeader
	OptionalHeader any // of type *OptionalHeader32 or *OptionalHeader64
	Sections       []*Section
	Symbols        []*Symbol    // COFF symbols with auxiliary symbol records removed
	COFFSymbols    []COFFSymbol // all COFF symbols (including auxiliary symbol records)
	StringTable    StringTable

	closer io.Closer
}

Unfortunately, in order to support BigObj COFF, (golang/go#24341), a few things need to change,.
BigObj files use a different type of header (ANON_OBJECT_HEADER_BIGOBJ) with largely the same fields as the FileHeader (IMAGE_FILE_HEADER) present already in the File struct.

You can find the C struct declarations in winnt.h.

The most important difference in the header is that FileHeader.NumberOfSections is now 32 bits, instead of 16 bits.
Supporting more sections in the file is the entire point of BigObj. I propose to add an extra:
int32 SectionCount field in the file struct to deal with this. In the case that the section count is greater than INT16_MAX, File.FileHeader.NumberOfSections will be zero. File.SectionCount will always contain an accurate number of sections.

File.FileHeader.NumberOfSections will be deprecated and new code should not rely on it, and instead refer to File.SectionCount to get the true number of sections.

The other fields in the header have 1:1 equivalents in the old FileHeader, so the code can just copy over the values from the BigObj header back to the FileHeader, and they can be accessed from that struct instead.

A similar approach has to be taken for Symbols and COFFSymbols.

Symbols inside regular COFF files are of the format IMAGE_SYMBOL (also present in winnt.h).
BigObj files instead have the format IMAGE_SYMBOL_EX. The only difference between them is that the SectionNumber field got expanded to 32 bits.
Unfortunately, the current COFFSymbol and Symbol structs have a 16 bit SectionNumber.

I propose adding two new fields:

BigObjSymbols []*BigObjSymbol
BigObjCOFFSymbols []BigObjCOFFSymbol

The only change in these structs will be a 32 bit SectionNumber.

Similar to the header, the old structs will be deprecated, and both of those arrays will be empty in the case that the section count is greater than INT16_MAX. New code should always prefer to use these fields.

For more details on BigObj, I wrote up a blog post explaining my findings when writing my initial patch:
https://peter0x44.github.io/posts/bigobj_format_explained/
Unfortunately, Microsoft doesn't document the binary format. And it seems no one else did either. So you just have to "Trust Me, Bro".
#75631

My patch is functional. The issues are that it breaks the go compatibility guarantee.

I am happy to bikeshed over the naming of the structs in this API. I couldn't come up with any I fully liked.

[Peter0x44]

cc @ianlancetaylor

[gabyhelp]

Related Issues

• debug/pe: add APIs to support reading COMDAT info for sections #51868 (closed)
• debug/pe: big-obj is not supported #24341

Related Code Changes

• debug/pe: add support for bigobj COFF format
• debug/pe: introduce File.COFFSymbols and (*COFFSymbol).FullName

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[ianlancetaylor]

CC @golang/windows

There are two parts to this proposal.

The first part is to add a new field to pe.File to hold the 32-bit number of sections. Currently pe.File embeds pe.FileHeader, and pe.FileHeader has the field NumberOfSections uint16. The proposal is to add a new field to pe.File (not pe.FileHeader): SectionCount uint32. But actually I'm not sure we need that field. I think that the field is redundant with the length of the Sections field. I think that we can instead tell people to use len(f.Sections) rather than f.NumberOfSections to get the number of sections.

Unfortunately, I don't know that we have a way to formally deprecate a field inherited from an embedded struct. There is nothing wrong with the FileHeader.NumberOfSections field; that field is correct in the cases where the FileHeader struct is correct. The problem is with the File struct, which should be able to represent both traditional PE files and BigObj files.

The second part of the proposal is to handle the section number of a symbol. Currently the type Symbol has a field SectionNumber int16. The proposal above suggests adding new BigObjSymbol and BigObjCoffSymbol types.

I don't think we need BigObjSymbol; the Symbol type does not correspond to any data structure in a PE file. I think we can add a new field to Symbol: BigSectionNumber uint32. Then we can deprecate SectionNumber in favor of that field.

It's less clear if we can do this for COFFSymbol. That type does currently correspond to a type in the file. Still, we could decide to break that correspondence and add a BigSectionNumber field to that type as well. And, for that matter, to COFFSymbolAuxFormat5, which is returned by the (*File).COFFSymbolReadSectionDefAux method. This would complicate the code in the debug/pe package, but would simplify the code that uses it. It would be unfortunate if users would have to switch from the Symbols slice to a BigObjSymbols slice. It would be especially unfortunate if they have to always switch. Bad enough to force people to switch from SectionNumber to BigSectionNumber.

[aclements]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— aclements for the proposal review group