Strange Visual Basic Error

[Yahweh]

From the thread titled Correa Neto's thread Anyone has acces to the software used to "decrypt" the Bible code? (http://www.randi.org/vbulletin/showthread.php?s=&threadid=50512), I thought it would be fun to build my own little decrypting tool (I was going to look for hidden messages found in Randi's commentary to prove he is in cohorts with the Raelians - or something fun like that).

I've been working for a very short time, so I have the skeleton of the kind of program I had intended:
* a textbox where source text is pasted
* another textbox to specify a search string
* a button to search sequence

I do not have access to the latest version of Visual Basic, so I have to settle for VBA provided by Excel 97 (yeah, I'm real oldschool).

Here is what I intended my code to do: I want to strip out all non-alphanumeric. Any characters that are neither A-Z, a-z, or 0-9 are to be stripped from the textbox. (Usually, I would just use the built in REPLACE function, however because I'm so oldschool my version of VB does not have that function, which forces me to code it myself. Its a small chore.)

Here is the code:
'Removing all non-alphanumeric characters from Textbox1 and strSearch
strSearch = UCase(strSearch) 'Making all text UPPERCASE for uniformity
TextBox1.Text = UCase(TextBox1.Text) 'Textbox1 contains source text
Caption = "Removing White Space - 0% Done" 'Nifty Progress Indicator Thingy
For I = 1 To 255
Select Case Chr(I)
Case "0" To "9":
Case "A" To "Z":
Case "a" To "z":
Case Else:
strSearch = Replace(strSearch, "", Chr(I))
TextBox1.Text = Replace(TextBox1.Text, "", Chr(I))
End Select

'Updating Nifty Process Indicator Thingy
Caption = "Removing White Space - " & Left((I / 255) * 100, 4) & "% Done"
Next I
'For some reason, the code does not remove lines, so I have to
'repeat this process manually
strSearch = Replace(strSearch, "", vbCrLf)
TextBox1.Text = Replace(TextBox1.Text, "", vbCrLf)

The Replace is a public function, here is the code for it:
Public Function Replace(ByVal strString As String, ByVal _
strReplaceWith As String, ByVal strReplace As String) As String

'I have an obsolete version of Visual Basic, so I have to manually
'program a Replace routine.
'
'This is not original code, it was find at:
'http://www.andreavb.com/tip110003.html

'************************************************* ************
'* Replicates the Replace Function found in Visual Basic 6.0
'* Usage: For VBA programming Word Macro etc
'* MyVar = Replace("Greg Hello World Greg", "David", "Greg")
'************************************************* ************
Dim I As Long
Dim strTemp As String

I = 1
Do While InStr(I, strString, strReplace, vbTextCompare) <> 0
strTemp = strTemp & Mid(strString, I, InStr(I, _
strString, strReplace, vbTextCompare) - I) & strReplaceWith
I = InStr(I, strString, strReplace, vbTextCompare) + Len(strReplace)
Loop
strTemp = strTemp & Right(strString, Len(strString) - I + 1)
Replace = strTemp
End Function


I will run the following phrase through my Non-alphanumeric filter:
0 1 2 3 4 5 6 7 8 9 0 - + = | \ The quick brown fox jumped over the lazy dog;:'"[{}]

abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ

Here is what the filter does to the text (the string below has no spaces or line breaks, however because it is a long string vBulletin is formatting it with spaces and breaks so it will fit on screen):
01234567890HEQUICKBROWNFOXJUMPEDOVERHELAZYDOGABCDE FGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZ

But that isnt correct. The "T" in each occurrence of the word "THE" is missing from the formatted string. However, in the alphabet, the T is present.

Through my own tedious analysis, I've determined that the code is inadvertantly mistaking the "TH" combination for the Þ (ASCII 222) character.

I could always add the Þ character to the list of exceptions, however that would disrupt the program if, for a strange reason, that character found a way into the source text. That method is not very practical in itself, because I still do not know what other character combinations could cause a problem, and it would be nearly impossible for me to find those combinations and what they are being mistaken for (it took about 15 minutes to find out that the Þ character was the one being mistaken).

The way I have my code set up seems to be the fastest and most efficient way to get the job done, I dont want to have to rewrite my code with an overcomplicated long procedure to remove non-alphanumeric characters.

Is there any explanation for this, and more importantly is there any practical way to fix this?

[SezMe]

Stand by while I give it a try ....

[Dogbreath]

Since you know what characters you want to keep, you may be better off looking for those and restringing them together. When you loop through all the other characters, some may do things you do not want like backspace, tab. Here is the code I modified just to look for the alphnumerics. You may want to turn it into a function so that you can call it for multiple strings.


'Removing all non-alphanumeric characters from Textbox1 and strSearch

Dim strOutput As String
strOutput = ""
strSearch = UCase(strSearch) 'Making all text UPPERCASE for uniformity
Textbox1.Text = UCase(Textbox1.Text) 'Textbox1 contains source text
Caption = "Removing White Space - 0% Done" 'Nifty Progress Indicator Thingy
For I = 1 To Len(Textbox1.Text)
Select Case Mid(Textbox1.Text, I, 1)
Case "0" To "9":
strOutput = strOutput & Mid(Textbox1.Text, I, 1)
Case "A" To "Z":
strOutput = strOutput & Mid(Textbox1.Text, I, 1)
Case "a" To "z":
strOutput = strOutput & Mid(Textbox1.Text, I, 1)
Case Else:
'strSearch = Replace(strSearch, "", Chr(I))
'Textbox1.Text = Replace(Textbox1.Text, "", Chr(I))
End Select
Debug.Print strOutput
'Updating Nifty Process Indicator Thingy
Caption = "Removing White Space - " & Left((I / 255) * 100, 4) & "% Done"
Next I
Textbox1.Text = strOutput

Here is the result. (it is one long string)
01234567890THEQUICKBROWNFOXJUMPEDOVERTHELAZYDOGABC DEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZ


I hope this helps.

[Yahweh]

Thank you very much, Dogbreath, it did exactly what it was supposed to do :)

However, to process 5000 characters on my very slow computer takes roughly 38 seconds.

Fortunately, about 10 minutes ago I noticed a quirk on my program in the Replace function: I am using the "vbTextCompare" method. I found by changing it to "vbBinaryCompare", the strange Þ/"TH" confusion goes away (by the way, I think that a problem with Visual Basic compiler, and not with the code itself). The code in its final form looks like this (changes are highlighted in red):
Public Function Replace(ByVal strString As String, ByVal _
strReplaceWith As String, ByVal strReplace As String) As String

'I have an obsolete version of Visual Basic, so I have to manually
'program a Replace routine.
'
'This is not original code, it was find at:
'http://www.andreavb.com/tip110003.html

'*************************************************
************
'* Replicates the Replace Function found in Visual Basic 6.0
'* Usage: For VBA programming Word Macro etc
'* MyVar = Replace("Greg Hello World Greg", "David", "Greg")
'*************************************************
************
Dim I As Long
Dim strTemp As String

I = 1
Do While InStr(I, strString, strReplace, vbTextCompare) <> 0
strTemp = strTemp & Mid(strString, I, InStr(I, _
strString, strReplace, vbBinaryCompare) - I) & strReplaceWith
I = InStr(I, strString, strReplace, vbBinaryCompare) + Len(strReplace)
Loop
strTemp = strTemp & Right(strString, Len(strString) - I + 1)
Replace = strTemp
End Function

By changing it to vbBinaryCompare, I can process 5000 characters is roughly 8 seconds (which is about par for a 5 year old computer).

But still, thank you for your suggestions, I appreciate it :)

[BobK]

I think this is about as quick as you can do it in VB6.

Concatenating strings is slower than insertion into a pre-allocated space.

Did a 10K text file in less than 1 sec on my 500mhz machine.
Creates one long string with no line breaks.

Here's the code. I put it all in the form. Move the code if necessary to access it from elsewhere.


Option Explicit

Private Sub Command1_Click()
Dim charlist As String, text As String
Dim filename As String

'if you want to load a text file, use the next two lines
filename = App.Path & "/" & "tourneyhistories.hhf" 'text file I used
LoadTextFile filename

charlist = CharacterSet 'string of acceptable characters

text = Text1.text

text = UCase(text) 'make all upper case
'remove if preserving case
Text1.text = ValidCharacters(text, charlist)

End Sub

Private Function CharacterSet() As String
'build your valid character set here
Dim x As Long, s As String

For x = 48 To 57 '0 to 9
s = s & Chr(x)
Next

For x = 65 To 90 'A to Z
s = s & Chr(x)
Next
'if you want to preserve case add
'loop 97 to 122 here

CharacterSet = s
End Function

Private Function ValidCharacters(text As String, charlist As String) As String
Dim x As Long, char As String * 1
Dim offset As Long, newstring As String

newstring = String(Len(text), " ") 'pre-allocate space for new string
'much faster than concatenating strings

Do Until x = Len(text)

x = x + 1
char = Mid(text, x) 'grab each char of original string
'one at a time
If InStr(charlist, char) Then
offset = offset + 1 'valid character found
Mid(newstring, offset) = char 'insert in new string
End If

Loop

ValidCharacters = RTrim(newstring) 'trim any excess spaces

End Function

Private Sub LoadTextFile(file As String)
Dim buff As Long, text As String

buff = FreeFile
Open file For Input As buff

text = Input(LOF(buff), buff)
Text1.text = text

Close buff

End Sub

[scribble]

You people could be very powerful if you started using "real" programming languages.

[BobK]

Actually Scribble, it's the programmers, not the language.

Scratch what I posted above. It's cr*p.

This will do a million character string in less than 3 seconds on my 500 mhz machine.

'uses a form with a textbox and command button
'use a Rich textbox for files larger than 32K


Option Explicit
Private Sub Command1_Click()

Dim charlist(255) As Boolean
Dim filename As String, text As String

'text file I used contains 30% invalid characters
'if you want to load the textbox from a text file,
'use the next two lines inserting your filename, otherwise rem them out

filename = App.Path & "/" & "tourneyhistories.hhf"
LoadTextFile filename

GetCharacterList charlist()

text = Text1.text

text = UCase(text) 'make all upper case
'rem out line if preserving case

Text1.text = ValidChars(text, charlist())

End Sub

Private Sub LoadTextFile(file As String)

Dim buff As Long, text As String

buff = FreeFile
Open file For Input As buff

text = Input(LOF(buff), buff)
Text1.text = text

Close buff

End Sub

Private Sub GetCharacterList(charlist() As Boolean)
'initialize the ascii values you want to keep to True

Dim x As Long

For x = 48 To 57 '0 to 9
charlist(x) = True
Next

For x = 65 To 90 'A to Z
charlist(x) = True
Next

'add another loop here for lower case if desired.

End Sub

Private Function ValidChars(text As String, charlist() As Boolean) As String

'returns a string containing only valid characters

Dim x As Long, y As Long, length As Long
Dim offset As Long, newstring As String

length = Len(text)

ReDim bstr(length - 1) As Byte 'new byte array the size of original text

Do Until x = length

x = x + 1

y = Asc(Mid(text, x, 1)) 'grab ascii value of each char of original string
'one at a time
If charlist(y) = True Then
bstr(offset) = y 'if True then save ascii value to byte array
offset = offset + 1

End If

Loop

newstring = StrConv(bstr(), vbUnicode) 'convert byte array to string

ValidChars = newstring

End Function

[SezMe]

Originally posted by scribble
You people could be very powerful if you started using "real" programming languages.
I thought that childish, yours-is-a-language-for-girlie-men crap went out with the PC/Mac wars.

Sheesh

[Zep]

BobK,

Just as a possible speed-up even further, what happens if you convert the original string to a byte array first, then do the entire manipulation as integer arithmetic and array processing, then convert the resulting array back to a string at the end? That is, do NO string "processing" at all?

Let me know if you decide to try it - I'd be interested in the comparative performance results.

cheers
zep

[Zep]

Originally posted by SezMe
I thought that childish, yours-is-a-language-for-girlie-men crap went out with the PC/Mac wars.

Sheesh I have ACTUALLY programmed computers using front-panel switches. And plug-cables. Long before Macs and PCs even existed.

You wimp.



:D

[69dodge]

Just for comparison, here it is in C++.
void alphanumericize(string &s)
{ s.resize(remove_if(s.begin(),s.end(),isalnum) - s.begin()); }(I leave it to the macho Zep to show us how to do it using front-panel switches and plug-cables. :D)

[BobK]

Zep,

I thought about that.

VB6 evidently manipulates all strings as unicode.

Although I can assign a string to a byte array such as...

text = "ABCD"
dim bytearray() as byte
bytearray = text

Text has 4 characters but a byte length of 8, due to being a unicode string. Len(text) = 4, LenB(text) = 8

Bytearray will end up storing the unicode version and have bounds of 0 to 7. If a foreign character in the text actually required two bytes I'd need a special case comparison.

Although there may be a way to put the bytes in the actual ansi string into a byte array, it seems easiet to let VB handle the unicode conversion and use Asc(mid(text, x, 1)) for comparison purposes.
I only do it once for each character in the text, so there doesn't seem to be much sense in storing the ascii values in a byte array before making the comparison.

If you know of a more efficient way to do it in VB I would be interested.

[scribble]

Originally posted by SezMe
I thought that childish, yours-is-a-language-for-girlie-men crap went out with the PC/Mac wars.


The PC/Mac wars are over? No one told me. Do the folks at Apple know? It's clear no one told my peer group.

As long as languages compile code that "does the same thing" to vastly different bytecode, comparing the relative merits of languages is absolutely acceptable. It's insane to claim that all languages are equivalent, and it's ridiculous to claim that some languages aren't better than others for given tasks. It's *NOT* ridiculous by any stretch of the imagination to claim that VB is not a good language for processing large data sets. The text of the Bible does not reach "large" by many standards, but it *does* reach "large -- too large" by VB standards. Hell, last time I coded in VB (nearly ten years ago while employed at Microsoft) the typical control still had a 64k limit on the data it could hold. In all fairness this was probably due to their failure to abstract away the segment:offset addressing system the operating system was tied to.

I stand by my claim.

[69dodge]

Oops. I got it backwards. The code I posted removes the alphanumeric characters and leaves the nonalphanumerics.

Here's the right way to do it. I think. (Maybe I should test it first? Nah ...)void alphanumericize(string &s)
{ s.resize(remove_if(s.begin(),s.end(),not1(ptr_fun( isalnum))) - s.begin()); }

[69dodge]

That thorn thing is pretty weird. Why does Visual Basic do that? You don't live in Iceland, by any chance, do you, Yahweh? :D

http://www.evertype.com/standards/wynnyogh/thorn.html

http://www.plexoft.com/cgi-bin/thorn.cgi

A reasonable comment on language comparisons, by Stroustrup (from The Design and Evolution of C++ (http://www.amazon.com/exec/obidos/tg/detail/-/0201543303), p. 5):<blockquote>Several reviewers asked me to compare C++ to other languages. This I have decided against doing. Thereby, I have reaffirned a long-standing and strongly held view: Language comparisons are rarely meaningful and even less often fair. A good comparison of major programming languages requires more effort than most people are willing to spend, experience in a wide range of application areas, a rigid maintenance of a detached and impartial point of view, and a sense of fairness. I do not have the time, and as the designer of C+, my impartiality would never be fully credible.

I also worry about a phenomenon I have repeatedly observed in honest attempts at language comparisons. The authors try hard to be impartial, but are hopelessly biased by focusing on a single application, a single style of programming, or a single culture among programmers. Worse, when one language is significantly better known than others, a subtle shift in perspective occurs: Flaws in the well-known language are deemed minor and simple workarounds are presented, whereas similar flaws in other languages are deemed fundamental. Often, the workarounds commonly used in the less-well-known language are simply unknown to the people doing the comparison or deemed unsatisfactory because they would be unworkable in the more familiar language.</blockquote>And now, a perhaps unreasonable comment, by me: Whenever I see someone write<blockquote>IF something = TRUE THEN ...</blockquote>I always want to ask why they didn't write<blockquote>IF (something = TRUE) = TRUE THEN ...</blockquote>instead. :D If "something" is already a boolean, you can just write<blockquote>IF something THEN ...</blockquote>and be done with it.

[Zep]

Originally posted by 69dodge
I leave it to the macho Zep to show us how to do it using front-panel switches and plug-cables. :D Click click click...LOAD...RUN


Thank you, thank you.

:D

Like this one, for example. Incidentally, what's so special about this one in particluar?

http://histoire.info.online.fr/images/pdp11-unix.jpeg

[69dodge]

Originally posted by 69dodge
the designer of C+I hate the time limit on editing.Originally posted by Zep
Incidentally, what's so special about this one in particluar?No idea.

[Zep]

The people in the picture are Bell and Ritchie, about 1972 (I think). Anyway, they are shown designing a new operating system... I forget it's name - Onyx? EMACS? Eunuchs?




:D

[Zep]

Well, for what it's worth, here's my VB6 code effort using a byte array in place. It's a straight replacement for BobK's last function only in his code.

I was running it on a P4 2.4GHz at work, so it was more or less "instant" for about 64K of characters (obviously not a good performance comparison with the slower CPUs - this is a CPU-intensive task). I'd be interested if others can try it out on lesser machines...
Private Function ValidChars(txt As String, charlist() As Boolean) As String
Dim chars() as byte, i as integer, j as integer, delta as integer
delta = lenB("A")
chars() = txt
j = delta * -1
for i = 0 to ubound(chars) step delta
if charlist(chars(i)) and (j <> i) then
j = j + delta
chars(j) = chars(i)
end if
next
txt = chars()
ValidChars = mid$(txt, 1, (j + delta) / delta)
End Function

[Iconoclast]

Originally posted by Zep
Like this one, for example. Incidentally, what's so special about this one in particluar?
It shows a computer programmer with a friend.

[Zep]

Originally posted by Iconoclast
It shows a computer programmer with a friend. (Couldn't possibly. Computer programmers don't have friends. They don't even have lives.)

[rockoon]

Originally posted by scribble

As long as languages compile code that "does the same thing" to vastly different bytecode, comparing the relative merits of languages is absolutely acceptable. It's insane to claim that all languages are equivalent, and it's ridiculous to claim that some languages aren't better than others for given tasks. It's *NOT* ridiculous by any stretch of the imagination to claim that VB is not a good language for processing large data sets.

Your last experience with VB was 10 years ago.

perhaps.. Sit down and be quiet?

VB6 uses microsofts C2 compiler. Also used for their timely (1998) C++ compiler. Both C++ and VB6 at the time converted their code to the same meta-language which was thrown at the C2 compiler which produced the final object files before linking.

Now VB7 (aka VB.NET) uses the same framework as all .net compilers... where the same MSIL assembler (yet another meta language) is used for all the languages.

But thanks for offering your decade-old opinion.

[BobK]

Zep,

Thanks for posting the code.

Your routine is about 3 times as fast when running in the VB environment (8.5 sec. vs 26 sec.), and about 12 times as fast when run as an exe. (1 sec vs 12 sec.) That's for 10 million characters on my 500mhz PIII.

I had considered doing something similar, but decided against it.

Won't that routine run into problems if processing text that makes use of the other byte of the unicode character? i.e a non-zero byte. Foreign language maybe? That's why I opted to let VB handle the unicode conversion from string to get the ascii value.

I don't know much about unicode, but I figure that other byte is used in some cases.

One point that may be of interest. By changing your 2 byte integer variables to 4 byte long, it processes about 10-12% faster. 32 bit processors evidently handle 32 bit variables more efficiently than 16 bit variables. Or maybe it has to do with the way the compiler handles them.

Bob

[rockoon]

Originally posted by BobK

One point that may be of interest. By changing your 2 byte integer variables to 4 byte long, it processes about 10-12% faster. 32 bit processors evidently handle 32 bit variables more efficiently than 16 bit variables. Or maybe it has to do with the way the compiler handles them.

Bob

This is an artifact of the way intel adopted 32-bit registers. Rather than double the number of instructions in the instruction set (one set for 16-bit operations, one set for 32-bit) they chose to use a special prefix byte in the machine code/assembler to toggle the next instruction between 16-bit and 32-bit.

They adopted specific defaults such that when the processor is in 16-bit real mode (also v86 mode) the default instructions are 16-bit and when in 32-bit protected mode the default instructions are 32-bit.

So in the case of a 32-bit compiled code performing 16-bit math, this prefix byte exists in front of every instruction that operates on a 16-bit value.

While often the compiler can fetch both the prefix byte and the following instruction while previous instructions are still executing (hiding the performance penalty), sometimes it cannot.

This accounts for the performance penalty you see when using 16 bit math while in 32-bit mode. The same penalty will exist when doing 32-bit math in 16-bit mode.

[jimlintott]

***Warning I am not a programmer, just a hack.****

To strip all non-alphanumeric characters fom a text file, I came up with this little program written in PERL.

#!/usr/bin/perl
#Expects name of text file on command line
open (Text, $ARGV[0]);
while(*Text*){s/\W//g;print;}
close Text;

Note that the file is dereferenced using the diamond operator which appears as a tag to the web. You would have to replace the * with appropriate less than and greater than characters in the while statement. (Is there a way to escape these?)

It will process a sligtly over 10 meg text file in about 11.5 seconds when the output is redirected to a file. (900 mhz Athlon, Kernel 2.6)

[Zep]

Originally posted by BobK
Zep,

Thanks for posting the code.

I only hope it worked elsewhere!

Your routine is about 3 times as fast when running in the VB environment (8.5 sec. vs 26 sec.), and about 12 times as fast when run as an exe. (1 sec vs 12 sec.) That's for 10 million characters on my 500mhz PIII.

I had considered doing something similar, but decided against it.

OK, thanks for doing that. I'm not sure if your routine was compiled or not, so I don't know what to compare it with. Certainly it's better than 38 seconds for the same task.

Won't that routine run into problems if processing text that makes use of the other byte of the unicode character? i.e a non-zero byte. Foreign language maybe? That's why I opted to let VB handle the unicode conversion from string to get the ascii value.

Indeed it will have problems - this was merely a rapid prototype proof of concept using single-byte ASCII codes only (I should have been working!). However the code for Unicode is not a great variation on what's there already (just a line or two added, I think). If you wanted to follow through on this, that is.

I don't know much about unicode, but I figure that other byte is used in some cases.

Correct. One char = adjacent bytes in the array. I found they were char(0) when pure ASCII was used.

One point that may be of interest. By changing your 2 byte integer variables to 4 byte long, it processes about 10-12% faster. 32 bit processors evidently handle 32 bit variables more efficiently than 16 bit variables. Or maybe it has to do with the way the compiler handles them.

Bob

Sure! Give it a try and see. And Rockoon has posted more detail on the subject that I was not aware of either - thanks, R, noted.

And well done Bob for thinking like us old-time programmers. We have always contended that you can make your programs way more efficient quickly and for free by using obvious improvements, rather than by just throwing heaps of expensive grunt at them. Already you have made nearly a 30-fold improvement without having to use a P4 3.6GHz monster or similar to get it.

[a_unique_person]

Originally posted by Zep

And well done Bob for thinking like us old-time programmers. We have always contended that you can make your programs way more efficient quickly and for free by using obvious improvements, rather than by just throwing heaps of expensive grunt at them. Already you have made nearly a 30-fold improvement without having to use a P4 3.6GHz monster or similar to get it.


I'll pretend I didn't hear you say that, Zep. As the IBM reps used to say, there is no problem that can't be solved by throwing more iron at it.

[Zep]

Originally posted by a_unique_person
I'll pretend I didn't hear you say that, Zep. As the IBM reps used to say, there is no problem that can't be solved by throwing more iron at it. Just sign right HERE, sir. And HERE, sir. And HERE, sir. Wonderful. Here's your new computer, sir. Now where are the children you have just pawned? Sir, we need the children BEFORE you get the keys to it...

What was REALLY a sight to see was some nicely optimised code running on some really powerful silicon and fast spinning rocks. Makes the PCs these days look like...well...home game computers!


:D

[a_unique_person]

Originally posted by Zep
Just sign right HERE, sir. And HERE, sir. And HERE, sir. Wonderful. Here's your new computer, sir. Now where are the children you have just pawned? Sir, we need the children BEFORE you get the keys to it...

What was REALLY a sight to see was some nicely optimised code running on some really powerful silicon and fast spinning rocks. Makes the PCs these days look like...well...home game computers!


:D

I think it really was cheaper to get more computer than hire people who knew what they were doing.

[Zep]

Originally posted by a_unique_person
I think it really was cheaper to get more computer than hire people who knew what they were doing. Not from IBM it wasn't...