I have a question regarding boxing in c#.

Currently, I have the need for a map like the following:

Dictionary<string, FieldValue>

Where field value looks something like this (i'm leaving out obvious code):

struct FieldValue
public object DataValue {...}
public UnitType Units{...}

It has come to my attention that the majority of values I will be storing in this map will be value types, and I will be performing quite a few adds per execution of this program, so I would like to avoid some of the boxing penalty of storing value types in the DataValue parameter of the above object.

Initially I thought, since I know all values will be either a number, a string, or a list of either type, that I would simply create 4 different fields, one for each type and store an enumeration that indicated which field was being used. I don't really like this solution, so another idea I had was to do something like the following (all value types will be stored in doubles):

class DoubleReference
public double DoubleValue{...}

That way the DoubleReference object could be instantiated beforehand and I think, if I understand correctly how things work, that I would be able to avoid at least an unboxing operation by storing DoubleReference rather than a regular double in the map of generic object types.

My question is, does doing this make any sense? Will it gain me anything at all, or am I wasting my time here? Thanks in advance for any help.