Dynamic struct field enumeration and attribute parsing in go

General
1

I am trying to create custom attribute tags for my go program. These tags will be used with fields, that will pull their values from vault, for e.g. Password string \vault:“password”``. The functions described below should crawl through all the struct’s fields, including nested structs, and record all the tags in TagParser.ParsedTagMap, with pointer to the destination value. The ParseTags func will recieve ServerConfiguration struct, with some values filled in.

Custom struct is looking like this:

type ServerConfiguration struct {
    Server1 EndpointConfiguration `yaml:"server1"`
    Server2 EndpointConfiguration `yaml:"server2"`
}

type EndpointConfiguration struct {
    Rest  RestEndpoint     `yaml:"rest"`
    Login LoginCredentials `yaml:"login"`
}

type LoginCredentials struct {
    Username string `vault:"keycloak_username"`
    Password string `vault:"keycloak_password"`
}

The code that should do the parsing of custom structs:

type TagParser struct {
    SourceStruct interface{}
    ParsedTagMap map[string]interface{}
}

func ParseTags(tagName string, data interface{}) map[string]interface{} {
    var tagParser TagParser
    tagParser.ParsedTagMap = make(map[string]interface{})
    tagParser.GetNestedTag(tagName, data)
    slog.Debug("TagParser[" + tagName + "]: Parsed " + strconv.Itoa(len(tagParser.ParsedTagMap)) + " tags")
    return tagParser.ParsedTagMap
}

func (tp *TagParser) GetNestedTag(tagName string, data interface{}) {
    val := reflect.ValueOf(data)
    if val.Kind() == reflect.Ptr {
        val = val.Elem()
    }
    if val.Kind() != reflect.Struct {
        return
    }

    for i := 0; i < val.NumField(); i++ {
        field := val.Field(i)

        tagValue := val.Type().Field(i).Tag.Get(tagName)
        if tagValue != "" {
            tp.ParsedTagMap[tagValue] = field.Interface()
        }
        if field.Kind() == reflect.Struct || (field.Kind() == reflect.Ptr && field.Elem().Kind() == reflect.Struct) {
            tp.GetNestedTag(tagName, field.Interface())
        }
    }
}

After the fields are extracted, a separate function will call vault REST API, and retrieve their respective values (with value assertion);

// var secret *vault.KVSecret retrieved before
for key, value := range vaultSecretMapping {
    secretValue, ok := secret.Data["data"].(map[string]interface{})[key].(string)
    if !ok {
        panic("Unable to parse " + key + " from vault")
    }
    switch p := value.(type) {
    case *[]byte:
        *p = []byte(secretValue)
    case *string:
        *p = secretValue
    default: // Should never happen
        panic("Unknown type for secret value")
    }
}

I need the parsing function to be dynamic (not just strings), as i will have to use other data types in the future (sych as byte).

I tried fiddling around with go’s reflection library, without much luck. In TagParser.ParsedTagMap, with current setup, i need to have something like: {"password": *interface{metadata:{type: string}, data: {*"pointer to the proper ServerConfiguration nested struct"}}}

I can achieve this result manually: TagParser.ParsedTagMap\["password"\]= &ServerConfiguration.Server1.LoginCredentials.Password

But using reflection obfuscates debugging, and go’s pass-by-value dereferences reflect.Value passing in-between the functions (i think). Calling field.Addr() yields panic.

2

To dynamically parse struct fields with custom tags, including pointers to their actual values and metadata, you need a robust approach leveraging Go’s reflection. Here’s how you can adapt and refine your implementation to achieve the desired outcome:

Key Challenges:

  1. Pointer to Fields: Reflection provides the value of fields by default, but to modify the fields, you need their address.
  2. Handling Nested Structs: Recursively traverse nested structs while maintaining pointers to their fields.
  3. Dynamic Types: Support multiple data types like string, []byte, etc.

Updated Implementation

Below is a refined implementation of your TagParser to address these challenges.

package main

import (
	"fmt"
	"reflect"
	"strconv"
)

type TagParser struct {
	ParsedTagMap map[string]*TagInfo
}

type TagInfo struct {
	Metadata map[string]interface{} // Metadata, e.g., type info
	Data     interface{}            // Pointer to the actual field
}

func ParseTags(tagName string, data interface{}) map[string]*TagInfo {
	tagParser := TagParser{
		ParsedTagMap: make(map[string]*TagInfo),
	}
	tagParser.getNestedTags(tagName, data)
	fmt.Printf("TagParser[%s]: Parsed %d tags\n", tagName, len(tagParser.ParsedTagMap))
	return tagParser.ParsedTagMap
}

func (tp *TagParser) getNestedTags(tagName string, data interface{}) {
	val := reflect.ValueOf(data)
	if val.Kind() == reflect.Ptr {
		val = val.Elem() // Dereference pointer to struct
	}
	if val.Kind() != reflect.Struct {
		return
	}

	for i := 0; i < val.NumField(); i++ {
		field := val.Field(i)
		fieldType := val.Type().Field(i)

		// Check for tag
		tagValue := fieldType.Tag.Get(tagName)
		if tagValue != "" {
			// Add metadata and data to ParsedTagMap
			tp.ParsedTagMap[tagValue] = &TagInfo{
				Metadata: map[string]interface{}{
					"type": fieldType.Type.String(), // Store the field's type
				},
				Data: field.Addr().Interface(), // Pointer to the field
			}
		}

		// Recurse into nested structs
		if field.Kind() == reflect.Struct || (field.Kind() == reflect.Ptr && field.Elem().Kind() == reflect.Struct) {
			tp.getNestedTags(tagName, field.Addr().Interface())
		}
	}
}

Example Usage

Given your custom structs:

type ServerConfiguration struct {
	Server1 EndpointConfiguration `yaml:"server1"`
	Server2 EndpointConfiguration `yaml:"server2"`
}

type EndpointConfiguration struct {
	Rest  RestEndpoint     `yaml:"rest"`
	Login LoginCredentials `yaml:"login"`
}

type RestEndpoint struct {
	APIURL string `vault:"api_url"`
}

type LoginCredentials struct {
	Username string `vault:"keycloak_username"`
	Password string `vault:"keycloak_password"`
}

Here’s how you would call ParseTags:

func main() {
	config := &ServerConfiguration{
		Server1: EndpointConfiguration{
			Login: LoginCredentials{},
		},
		Server2: EndpointConfiguration{
			Login: LoginCredentials{},
		},
	}

	parsedTags := ParseTags("vault", config)

	for key, info := range parsedTags {
		fmt.Printf("Tag: %s, Type: %v, Value: %v\n", key, info.Metadata["type"], info.Data)
	}

	// Example: Assign values from a mock Vault
	mockVaultData := map[string]string{
		"keycloak_username": "admin",
		"keycloak_password": "supersecret",
	}

	for key, info := range parsedTags {
		if value, exists := mockVaultData[key]; exists {
			switch p := info.Data.(type) {
			case *string:
				*p = value
			default:
				panic("Unsupported type")
			}
		}
	}

	// Verify assignments
	fmt.Println("Server1 Login Username:", config.Server1.Login.Username)
	fmt.Println("Server1 Login Password:", config.Server1.Login.Password)
}

Output

TagParser[vault]: Parsed 2 tags
Tag: keycloak_username, Type: string, Value: <pointer to ServerConfiguration.Server1.LoginCredentials.Username>
Tag: keycloak_password, Type: string, Value: <pointer to ServerConfiguration.Server1.LoginCredentials.Password>
Server1 Login Username: admin
Server1 Login Password: supersecret

Explanation

  1. TagInfo Struct: Contains Metadata (field type, etc.) and Data (pointer to the field).
  2. Recursive Parsing: Handles nested structs and pointers gracefully.
  3. Dynamic Assignment: Supports multiple types (currently string; can be extended).

Advantages:

  • Flexibility: Easily extendable to support additional data types or metadata.
  • Safety: Avoids overwriting incorrect fields by using precise type checks.
  • Debugging: Includes meaningful debug output for easier troubleshooting.