Reversing-golang Reversing Go - Part 2
Post
Cancel

Reversing Go - Part 2

Recognizing common constructs

Append

Append is implemented using growslice

Case 1 - General Use

1
2
3
4
5
6
7
8
9
10
11
12
13
package main

import (
    "fmt"
    "os"
    "unsafe"
)

func main() {
    t := os.Environ()
    p := append(t, "A", "B", "C")
    fmt.Printf("t = %x\np = %x\n", unsafe.Pointer(&t), unsafe.Pointer(&p))
}

Now the compiler compiles it to

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
.text:004A7556    lea     rax, string_autogen_SNN2L2
.text:004A755D    mov     [rsp+90h+var_90], rax
.text:004A7561    call    runtime_newobject
.text:004A7566    mov     rax, [rsp+90h+var_88]
.text:004A756B    mov     [rsp+90h+newSlice], rax
.text:004A7570    call    syscall_Environ
.text:004A7575    mov     rax, [rsp+90h+var_90]
.text:004A7579    mov     rcx, [rsp+90h+var_88]
.text:004A757E    mov     rdx, [rsp+90h+var_80]
.text:004A7583    mov     rdi, [rsp+90h+newSlice]
.text:004A7588    mov     [rdi+slice.len], rcx
.text:004A758C    mov     [rdi+slice.cap], rdx
.text:004A7590    cmp     cs:runtime_writeBarrier, 0
.text:004A7597    jnz     loc_4A77A5
.text:004A759D    mov     [rdi+slice.data], rax

A new slice (3 words) is malloc’d using runtime.newobject and the return value os syscall.Environ is assigned to the newly created empty slice.

Next comes the call to append.

1
2
3
4
5
6
7
8
9
10
11
.text:004A75A0    lea     rax, string_autogen_SNN2L2
.text:004A75A7    mov     [rsp+90h+var_90], rax
.text:004A75AB    call    runtime_newobject
.text:004A75B0    mov     rdi, [rsp+90h+var_88]
.text:004A75B5    mov     rax, [rsp+90h+newSlice]
.text:004A75BA    mov     rcx, [rax+8]              ; newSlice.len
.text:004A75BE    mov     rdx, [rax+10h]            ; newSlice.cap
.text:004A75C2    mov     rbx, [rax]                ; newSlice.ptr
.text:004A75C5    lea     rsi, [rcx+3]
.text:004A75C9    cmp     rsi, rdx
.text:004A75CC    ja      need_more_space

It checks if the capacity of newSlice is enough to accomodate 3 more elements. If the capacity is smaller, a new slice is allocated using growslice

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
.text:004A775A    lea     rax, string_autogen_PMMZGP
.text:004A7761    mov     [rsp+90h+var_90], rax
.text:004A7765    mov     [rsp+90h+var_88], rbx
.text:004A776A    mov     [rsp+90h+var_80], rcx
.text:004A776F    mov     [rsp+90h+var_78], rdx
.text:004A7774    mov     [rsp+90h+var_70], rsi
.text:004A7779    call    runtime_growslice
.text:004A777E    mov     rbx, [rsp+90h+var_68] ; ptr
.text:004A7783    mov     rax, [rsp+90h+var_60] ; len
.text:004A7788    mov     rdx, [rsp+90h+var_58] ; cap
.text:004A778D    lea     rsi, [rax+3]
.text:004A7791    mov     rax, [rsp+90h+newSlice]
.text:004A7796    mov     rcx, [rsp+90h+var_40]
.text:004A779B    mov     rdi, [rsp+90h+var_38]
.text:004A77A0    jmp     append_elements

Now we know how to deduce params, so we can say that growslice has the signature

1
func growslice(tp *rtype, oldSlice slice, newCap int) slice
1
2
3
4
5
6
7
8
9
10
11
12
.text:004A75D2    shl     rcx, 4
.text:004A75D6    mov     qword ptr [rbx+rcx+8], 1  ; set length of new string
.text:004A75DF    lea     r8, [rbx+rcx]
.text:004A75E3    lea     r9, [rbx+rcx]
.text:004A75E7    lea     r9, [r9+10h]
.text:004A75EB    lea     r10, [rbx+rcx]
.text:004A75EF    lea     r10, [r10+20h]
.text:004A75F3    cmp     cs:runtime_writeBarrier, 0
.text:004A75FA    nop     word ptr [rax+rax+00h]
.text:004A7600    jnz     loc_4A7736
.text:004A7606    lea     r8, a5Abclmnpsz+0Dh ; "ABCLMNPSZ[\\\n\t"
.text:004A760D    mov     [rbx+rcx], r8

rbx points to the slice’s ptr, rcx contains the length of the slice. This snippet assigns the word at offset rcx*16+8 to 1 and the word at offset rcx*16+0 to the string “ABCL..”.

From Part-1 we know that a string has two words - ptr and len. So, here we are assigning a string of length 1 to the index stored in rcx. A string slice of n+1 elements looks something like this -

+00 ptr -> [ptr[0], len[0], ptr[1], len[1], ..., ptr[n], len[n]]
+08 len
+10 cap
1
2
3
4
5
.text:004A7611    mov     qword ptr [rbx+rcx+18h], 1
.text:004A761A    cmp     cs:runtime_writeBarrier, 0
.text:004A7621    jnz     loc_4A7719
.text:004A7627    lea     r8, a5Abclmnpsz+0Eh ; "BCLMNPSZ[\\\n\t"
.text:004A762E    mov     [rbx+rcx+10h], r8

At index rcx+1, it assigns the string “B”

1
2
3
4
5
.text:004A7633    mov     qword ptr [rbx+rcx+28h], 1
.text:004A763C    cmp     cs:runtime_writeBarrier, 0
.text:004A7643    jnz     loc_4A76FF
.text:004A7649    lea     r8, a5Abclmnpsz+0Fh ; "CLMNPSZ[\\\n\t"
.text:004A7650    mov     [rbx+rcx+20h], r8

At index rcx+2, it assigns the string “C”

Summarizing,

1
newSlice = append(newSlice, oldSlice...)

is compiled to

1
2
3
4
5
6
7
8
9
oldLen := len(newSlice)
newLen := oldLen+len(oldSlice)
if newLen > cap(newCap) {
    // allocate a larger slice
    newSlice = growslice(newSlice, newLen)
}
newSlice[oldLen] = oldSlice[0]
newSlice[oldLen+1] = oldSlice[1]
// ...

Case 2 - A special case of append

We know that make calls a specialized implementation on the type being used on. if we call make using a slice type, the compiler calls runtime.makeslice, for channels, it uses runtime.makechan and for maps, runtime.makechan

let’s consider the following snippet

1
2
// ...
a = append(a, make([]string, 1024)...)

Does go call makeslice and then growslice? No, it doesn’t. If you see the implementation of makeslice, you will find that it calls mallocgc with the third param (zero out the allocated memory) set to true. So, makeslice always returns a zeroed out slice. Now in this case, we are appending zeros to a. The compiler knows this, and it optimizes and removes the call to makeslice. Why?

append calls growslice which in turn calls mallocgc to allocate memory if the capacity is less. So, instead of two calls to mallocgc, only one call to mallocgc is required.

1
2
3
4
5
6
7
8
9
10
11
12
13
.text:004A707C    lea     rax, string_autogen_I57PDL
.text:004A7083    mov     [rsp+0A0h+var_A0], rax
.text:004A7087    call    runtime_newobject
.text:004A708C    mov     rdi, [rsp+0A0h+var_98]
.text:004A7091    mov     [rsp+0A0h+p], rdi
.text:004A7096    mov     rax, [rsp+0A0h+t]
.text:004A709B    mov     rcx, [rax]        ; ptr
.text:004A709E    mov     rdx, [rax+8]      ; len
.text:004A70A2    mov     rbx, [rax+10h]    ; cap
.text:004A70A6    lea     rsi, [rdx+1024]   ; just increase the length
.text:004A70AD    mov     [rsp+0A0h+var_48], rsi
.text:004A70B2    cmp     rsi, rbx
.text:004A70B5    ja      need_more_space       ; growslice

If growslice is not called, it zeroes out the part of the slice that needs to be appended - 1024*16 bytes starting from len(a).

1
2
3
4
5
6
7
8
9
10
.text:004A70C0    cmp     r8, rcx
.text:004A70C3    jz      clear_memory
; ...
clear_memory:
.text:004A7189    shl     rdx, 4
.text:004A718D    lea     rax, [rcx+rdx]
.text:004A7191    mov     [rsp+0A0h+var_A0], rax
.text:004A7195    mov     [rsp+0A0h+var_98], 4000h  ; clear 16*1024 bytes
.text:004A719E    xchg    ax, ax
.text:004A71A0    call    runtime_memclrHasPointers

From the docs,

memclrHasPointers clears n bytes of typed memory starting at ptr. The caller must ensure that the type of the object at ptr has pointers, usually by checking typ.ptrdata. However, ptr does not have to point to the start of the allocation.

This makes sense, since a string is composed of a pointer and it’s length.

Summarizing, we have

1
2
// ...
a = append(a, make([]string, N)...)

is implemented by

1
2
3
4
5
6
7
8
9
10
11
oldLen := len(a)
newLen := N+oldLen
ptr := &a[0]
if newLen > cap(a) {
    a = growslice(a, newLen)
}
if ptr == &a[0] {
    memset(a[oldLen:], N*sizeof(a[0]))
    // of course memset is not there in go
    // equivalent would be memclr family of functions
}

Strings, Bytes and Runes

Rune to String

1
a = string(rune(R))

is compiled to

1
a = intstring(nil, R)

Array of Runes/Bytes to string

for runes,

1
2
3
var t []rune
// ...
a = string(t)

compiles to

1
a = slicerunetostring(tmpBufPtr, t)

for bytes,

1
2
3
var t []byte
// ...
a = string(t)

compiles to

1
a = slicebytetostring(tmpBufPtr, t)

Now, tmpBufPtr is a pointer to an array of 32 bytes, if t does not escape to heap. If t escapes to heap, tmpBufPtr is nil

String to Byte array/Rune array

1
2
3
4
func main() {
    fmt.Println([]byte("I love Go"))
    fmt.Println([]byte(os.Args[0]))
}
1
2
3
4
5
6
7
8
9
10
11
.text:004A57C8    lea     rax, stru_4B2CE0  ; [9]byte
.text:004A57CF    mov     [rsp+68h+var_68], rax
.text:004A57D3    call    runtime_newobject
.text:004A57D8    mov     rax, [rsp+68h+var_60]
.text:004A57DD    mov     rcx, 'G evol I'
.text:004A57E7    mov     [rax], rcx
.text:004A57EA    mov     byte ptr [rax+8], 'o'
.text:004A57EE    mov     [rsp+68h+var_68], rax
.text:004A57F2    mov     [rsp+68h+var_60], 9
.text:004A57FB    mov     [rsp+68h+var_58], 9
.text:004A5804    call    runtime_convTslice

So, as you can see, an array of [9]byte is created using newobject and the string is copied into that array. I will explain the convTN family later.

1
2
3
4
5
6
7
8
9
10
.text:004A585F    mov     rcx, cs:os_Argc
.text:004A5866    mov     rax, cs:os_Args       ; []string
.text:004A586D    test    rcx, rcx
.text:004A5870    jbe     loc_4A5916
.text:004A5876    mov     rcx, [rax]            ; os.Args[0]
.text:004A5879    mov     rax, [rax+8]          ; os.Args[0].len
.text:004A587D    mov     [rsp+68h+var_68], 0   ; tmpBufPtr
.text:004A5885    mov     [rsp+68h+var_60], rcx ; str.ptr
.text:004A588A    mov     [rsp+68h+var_58], rax ; str.len
.text:004A588F    call    runtime_stringtoslicebyte

If the string is not a literal, then stringtoslicebyte is called to get a byte array, for runes, the corresponding function is stringtoslicerune. Here are the signatures of the functions

1
2
func stringtoslicebyte(*[32]byte, string) []byte
func stringtoslicerune(*[32]rune, string) []rune

convTN family of functions

from primitive types

convT16, convT32, convT64, convTstring, convTslice allocates the respective structures in heap and returns a pointer to it. convT16, convT32, convT64 are used for 16, 32 and 64 bit types.

When are these functions used? These functions are used when we try to convert a primitive data type to an interface{}

For example,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
func main() {
    var tv, iv interface{}
    iv = os.Args[0]
    fmt.Println(iv)
    iv = 0xcafe
    fmt.Println(iv)
    iv = "I love Go!"
    fmt.Println(iv)
    iv = []byte("I love Rust!")
    fmt.Println(iv)
    tv = []byte("I love Go and Rust!")
    fmt.Println(iv)
    iv = tv
    fmt.Println(iv)
}
1
2
3
4
5
6
7
8
9
10
11
.text:004A57F4    mov     [rsp+0C8h+var_C8], rax
.text:004A57F8    mov     [rsp+0C8h+var_C0], rcx
.text:004A57FD    nop     dword ptr [rax]
.text:004A5800    call    runtime_convTstring
.text:004A5805    mov     rax, [rsp+0C8h+var_B8]
.text:004A580A    xorps   xmm0, xmm0
.text:004A580D    movups  [rsp+0C8h+var_18], xmm0
; make eface
.text:004A5815    lea     rcx, string_autogen_CT9221    ; *string
.text:004A581C    mov     qword ptr [rsp+0C8h+var_18], rcx
.text:004A5824    mov     qword ptr [rsp+0C8h+var_18+8], rax

convTstring is used to get a pointer to os.Args[0] and construct the interface{} value, whose type is *string (pointer to string)

for literals, the interface is directly constructed using the address of the object

1
2
3
4
5
6
7
8
9
10
11
12
.text:004A5872    lea     rax, int
.text:004A5879    mov     qword ptr [rsp+0C8h+var_28], rax
.text:004A5881    lea     rax, qword_4E9EC0 ; 0xcafe
.text:004A5888    mov     qword ptr [rsp+0C8h+var_28+8], rax
; ...
.text:004A58D6    lea     rax, string_autogen_CT9221
.text:004A58DD    mov     qword ptr [rsp+0C8h+var_38], rax
.text:004A58E5    lea     rax, off_4EA330   ; *string
.text:004A58EC    mov     qword ptr [rsp+0C8h+var_38+8], rax
; ...
.rdata:004EA330 off_4EA330  dq offset aILoveGo  ; "I love Go!"
.rdata:004EA338             dq 0Ah

For slices, an array is constructed using newobject and then convTslice is used to construct an interface

1
2
3
4
5
6
7
8
9
10
11
12
13
14
.text:004A592F    lea     rax, stru_4B15C0  *[19]byte
.text:004A5936    mov     [rsp+0C8h+var_C8], rax
.text:004A59E5    call    runtime_newobject
.text:004A59EA    mov     rax, [rsp+0C8h+var_C0]
.text:004A59EF    mov     rcx, 'G evol I'
.text:004A59F9    mov     [rax], rcx
.text:004A59FC    mov     rcx, 'a oG evo'
.text:004A5A06    mov     [rax+3], rcx
.text:004A5A0A    mov     rcx, '!tsuR dn'
.text:004A5A14    mov     [rax+0Bh], rcx
.text:004A5A18    mov     [rsp+0C8h+var_C8], rax
.text:004A5A1C    mov     [rsp+0C8h+var_C0], 13h
.text:004A5A25    mov     [rsp+0C8h+var_B8], 13h
.text:004A5A2E    call    runtime_convTslice

from non-primitive types

Let’s consider the following snippet

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
type I1 interface {
	Method1()
}

type I2 interface {
	Method1()
	Method2()
}

type S struct {
	x, y int64
}

func (S) Method1() {}
func (S) Method2() {}

func main() {
	var e interface{}
	var s S
	var i1 I1
	var i2 I2
	e = s           // convT2E
	fmt.Println(e)
	i1 = s          // convT2I
	e = i1          // no conversion
	fmt.Println(e)
	i2 = s          // convT2I
	fmt.Println(i2)
	i1 = i2         // convI2I
	fmt.Println(e)
}

For e = s, we have the following,

1
2
3
4
5
6
7
.text:004A5976    xorps   xmm0, xmm0
.text:004A5979    movups  [rsp+0B8h+var_70], xmm0
.text:004A597E    lea     rax, main_S
.text:004A5985    mov     [rsp+0B8h+var_B8], rax
.text:004A5989    lea     rax, [rsp+0B8h+var_70]
.text:004A598E    mov     [rsp+0B8h+var_B0], rax
.text:004A5993    call    runtime_convT2Enoptr

when we try to assign a type T (which is not a 64 bit word or a slice or a string) to an interface, convT2E and convT2Enoptr is used. If T has embedded pointers, convT2E is used. It constructs an eface (empty interface) instance from type T.

1
func convT2E(t *_type, elem unsafe.Pointer) (e eface)

Now what if the target of the assignment is a iface (non empty interface, eface is an empty interface). Then convT2I is used.

Consider the statement i1 = s in the above code, we are assigning a struct instance to a non empty interface, in this case, convT2I is used

1
2
3
4
5
6
7
.text:004A59F8    xorps   xmm0, xmm0
.text:004A59FB    movups  [rsp+0B8h+var_70], xmm0
.text:004A5A00    lea     rax, go_itab_main_S_main_I1
.text:004A5A07    mov     [rsp+0B8h+var_B8], rax
.text:004A5A0B    lea     rax, [rsp+0B8h+var_70]
.text:004A5A10    mov     [rsp+0B8h+var_B0], rax
.text:004A5A15    call    runtime_convT2Inoptr

convT2I converts type T to a non empty interface (an interface with a valid set of functions)

1
2
3
4
5
6
7
8
9
10
11
12
13
type iface struct {
	tab  *itab
	data unsafe.Pointer
}

type eface struct {
	utype *_type
	data  unsafe.Pointer
}

func convT2E(t *_type, elem unsafe.Pointer) (e eface)
func convT2I(tab *itab, elem unsafe.Pointer) (i iface)
func convI2I(inter *interfacetype, i iface) (r iface)

the itab structure is discussed in part-1

For the statement i1 = i2, we can do that because the methods exposed by interface I1 are contained in the interface I2. The go compiler uses convI2I for this scenario

1
2
3
4
5
6
7
.text:004A5B28    lea     rax, main_I1  ; *interfaceType
.text:004A5B2F    mov     [rsp+0B8h+var_B8], rax
.text:004A5B33    mov     rax, [rsp+0B8h+var_78]    ; iface.utype
.text:004A5B38    mov     [rsp+0B8h+var_B0], rax
.text:004A5B3D    mov     rax, [rsp+0B8h+var_60]    ; iface.data
.text:004A5B42    mov     [rsp+0B8h+var_A8], rax
.text:004A5B47    call    runtime_convI2I

convI2I takes the target interfaceType, and the interface (iface) we want to convert and returns the iface pointing to the target type. How does convI2I do it?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
func convI2I(inter *interfacetype, i iface) (r iface) {
	tab := i.tab
	if tab == nil {
		return
	}
	if tab.inter == inter {
		r.tab = tab
		r.data = i.data
		return
	}
	r.tab = getitab(inter, tab._type, false)
	r.data = i.data
	return
}

If we are assigning interfaces whose types are same, like some instance of I1 to I1, then the itab table is retained. Otherwise, getitab searches the global table of itabs (itabTable) for the interface type we want to convert to (i1’s type) and the underlying type as the type we are converting from (i2’s underlying type).

In this example, underlying type of i2 is the struct S, and the interface type of i1 is the interfaceType I1. getitab searches for an itab with interface type I1 and underlying type S and returns a pointer to it. It must return itab_S_I1 since this is the itab that satisfies the conditions

What did we learn?

  1. append function
  2. type conversions
This post is licensed under CC BY 4.0 by the author.