VB.Net and Binary files

[Russell P. Holsclaw]

In the discussions about .NET, there has been scant mention of the effect of all this on binary files, that is, files opened with an "Open filename For Binary as filenumber", and read/written using the Get and Put statements.

It happens I use this extensively in my code. For example, I've got a great big hairy sparse matrix thingee called a "Judgment Base Matrix" (it's a type of fuzzy-logic AI Knowledge base). I load and save this monstrosity of data into a binary file using Get and Put statements, because that turned out to be the fastest way of doing it. Using an ASCII file required several minutes to Load/Save. In binary, it becomes a few seconds. Think of it like a Spreadsheet, or some other such complex collection of data structures that is handled as a "document".

One of the biggest time-savers in the Load/Save performance picture is a Put and Get of a Variant array, in which each element of the array contains an array of integers (the two-byte kind!). The total array of arrays takes up most of the file, typically from 500KB to 1MB in size. It takes a single Put to write it out, and a single Get to read it.

Now, will it ever be possible to write a VB.NET program that can read this file? It seems unlikely. What am I to do?

What kind of support will binary files have in VB.NET, and what kind of performance can I expect?

By the way, the file also contains some UDTs read and written using Get and Put, and they each contain a mix of Integers and Longs, which are presumed to be 2 and 4 bytes long, respectively. (ahem!) There's some fixed-length strings in some of the UDTs, also.

This file structure has migrated without a glitch from one version of VB to another since VB4. The end of the road seems to be approaching.

[Tim Jones]

Russell

> In the discussions about .NET, there has been scant mention of the effect of all this on binary files, that is, files opened with an "Open filename For Binary as filenumber", and read/written using the Get and Put statements.

I'm *VERY* interested in this as well, as it will affect a lot of
our stuff as well!

--
Regards, Tim Jones, http://www.aquatee.com

[Bob Butler]

"Russell P. Holsclaw" <rholsclaw@rentaudio.com> wrote in message
news:39fa0270$1@news.devx.com...
>In the discussions about .NET, there has been scant mention of the effect
of all this
>on binary files, that is, files opened with an "Open filename For Binary as
filenumber",
>and read/written using the Get and Put statements.
<cut>

Get/Put are listed in the help and show up in the assemblies... I did a
little toying with it and got it to work somewhat. It did not like arrays
but I think that can be solved with explicit casting. FYI, this runs for
me:

Imports System
Imports Microsoft.VisualBasic
Imports Microsoft.VisualBasic.Compatibility.VB6

Namespace JustATest

Public Class TheTest

Shared Sub Main()
Dim o As New TheTest
o.Doit()
o=Nothing
End Sub

Public Sub Doit()
Dim f As Int32
Dim x As Int32
Dim bData(256) As byte

f=FreeFile()
Open(f, "test.bin", OpenMode.Binary, OpenAccess.ReadWrite)
For x=0 To 255
Put(f,CByte(255-x))
Next x
Close(f)

Open(f, "test.bin", OpenMode.Binary, OpenAccess.ReadWrite)
For x=0 To 255
Compatibility.VB6.Get(f,bData(x))
Next x
Close(f)
Msgbox("Byte 100 is: " & cstr(bData(100)))

End Sub

End Class

End Namespace

[Russell P. Holsclaw]

Bob Butler <butlerbob@earthlink.net> wrote in message
news:39fa47bf@news.devx.com...
> "Russell P. Holsclaw" <rholsclaw@rentaudio.com> wrote in message
> news:39fa0270$1@news.devx.com...
> >In the discussions about .NET, there has been scant mention of the effect
> of all this
> >on binary files, that is, files opened with an "Open filename For Binary
as
> filenumber",
> >and read/written using the Get and Put statements.
> <cut>
>
> Get/Put are listed in the help and show up in the assemblies... I did a
> little toying with it and got it to work somewhat. It did not like arrays
> but I think that can be solved with explicit casting. FYI, this runs for
> me:
><snip the code>



OK, Bob, this works on a fixed-size byte array, but it's the simplest kind
of array there is.

But my example was about an array of variants, with each element containing
an array of integers. In this case, VBs 4 through 6 wrote out each element
of the variant array as a variant type value (array of integers), followed
by an "array descriptor", which provides the boundary data for the array,
followed by the contents of the array itself. What it writes to the file is
a continuous stream of data taken from non-contiguous areas of memory. This
operation is more than a mere dumping out of an area of memory, but a type
of "gather-write" operation (as we used to say in the mainframe world).
Interestingly, it runs about 50 times faster than any attempt on my part to
write code to explicitly write each element of each array individually.
Naturally, I chose the one that allows my file to be saved in a few seconds
instead of approx. 10 minutes.

In order to read this file in VB7, I'm going to first have to scope out the
variant and array descriptors and interpret them explicitly. I'm not at all
optimistic that if I do that, I will still be able to load and save the file
in a few seconds, like I am today.

[Bob Butler]

"Russell P. Holsclaw" <rholsclaw@rentaudio.com> wrote in message
news:39fd9c2c$1@news.devx.com...
>
<cut>
> OK, Bob, this works on a fixed-size byte array, but it's the simplest kind
> of array there is.
<cut>

Understood, and my point was only that binary mode, Get and Put still
existed. I have not yet figured out if they will work with the arrays like
you use but haven't had a lot of time to play with it either. If it can't
be done then I'm not going to like it either as I use binary access w/arrays
in a number of places for similar performance reasons. I'm hoping MSFT will
chime in with more info about it.

[Shelly Rosenfeld]

Don't mean to burst your bubble, but you ARE aware that
VB7 does not support the variant type...

Sheldon

Russell P. Holsclaw <rholsclaw@rentaudio.com> wrote in message
news:39fd9c2c$1@news.devx.com...
>
> But my example was about an array of variants, with each element
containing
> an array of integers. In this case, VBs 4 through 6 wrote out each
element
> of the variant array as a variant type value (array of integers), followed
> by an "array descriptor", which provides the boundary data for the array,
> followed by the contents of the array itself. What it writes to the file
is
> a continuous stream of data taken from non-contiguous areas of memory.

[Russell P. Holsclaw]

> Don't mean to burst your bubble, but you ARE aware that
> VB7 does not support the variant type...

Of course I am. That's why I mentioned this case. It's another
illustration of the
impact of the changes being made with .NET, and how "conversion utilities"
are limited
in their capability.

I think the transition to .NET could be accomplished more readily by most
developers if MS devoted some time to these migration issues.

As I understand it, the Variant is being replaced by a more flexible object
model. That being the case, it is not inconceivable that the new object
type that replaces variant by actually be "variant compatible" when it comes
to binary file I/O. I hadn't seen anything about MS actually doing
something like that, but it doesn't hurt to wish. Maybe someone in MS will
read this and respond.

Or, in the end, this will just give rise to a market niche for an
enterprising third-party developer.

[David Bayley]

Russel,

Following is the QuickStart\howto\Serialization sample that uses the
BinaryFormatter class. I think it will provide the functionality you need,
although I'm not sure. The docs say...

"Provides a way to serialize and deserialize an object, or an entire graph
of connected objects; it uses a binary format for the serialized stream,
which is both very compact, and fast to parse."

....where object can be read Variant.

<code>
Imports System
Imports System.IO
Imports System.Collections
Imports System.Runtime.Serialization
Imports System.Runtime.Serialization.Formatters.Binary

Namespace ClassLibrary1
Public Class SerializeTest
Public Shared Sub Main()
Console.WriteLine("Create object graph")
Dim l As New ArrayList
Dim x As Integer
For x = 0 To 9
Console.WriteLine(x)
l.Add(x)
Next x
Console.Write("Serializing object graph to disk..")
Dim s As Stream = (New File("foo.bin")).Open(FileMode.Create)
Dim b As BinaryFormatter = New BinaryFormatter
b.Serialize(s, l)
s.Close()
Console.WriteLine("Complete.")

Console.Write("Deserializing object graph from disk..")
Dim r As Stream = (New File("foo.bin")).Open(FileMode.Open)
Dim c As New BinaryFormatter
Dim p As ArrayList = CType(c.Deserialize(r), ArrayList)
Console.WriteLine("Complete.")
Dim i As Object
For Each i In p
Console.WriteLine(i)
Next i

r.Close()

Console.WriteLine("\r\nPress Return to exit.")
Console.Read()
End Sub
End Class
End Namespace
</code>

--
David.