...life can be translucent

Menu

A partial solution to the garbled characters in the archives

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
The problem of strings in the text, like & # 34 ; which make archived posts difficult to read. (Ignore the spaces - I had to put them there so the forum software wouldn't convert the codes to symbols. The "code" button didn't help for some reason.)

I just found a Firefox add-on which fixes this. It's called FoxReplace, and can be added to your Firefox from the link below. (Googling "chrome add on replace characters in a webpage" produces a couple likely-looking results, but I don't use Chrome and have not tried them. Does I.E. have add-ons? A quick glance at a similar Google query didn't look as promising.)

Link to FoxReplace:

https://addons.mozilla.org/En-us/firefox/addon/foxreplace/

It seems pretty easy to use. Just a few minutes of work seems to have cleared up most of the garbles. I haven't tried optional features like "substitution groups," but it works fine without.

One useful feature which seems not to work, at least for me, is the keyboard shortcut. The add-on claims that Shift-F8 will apply substitutions to the current webpage, but when I try it I get a window labelled "Firefox WebIDE," whatever that is. So I just use the menu.

Here are the substitutions I've entered so far (again, ignore the spaces):


style="width: 500px" class="cms_table_grid" align="center"
|-
| Code
| Symbol
| Description
|-
| & # 34 ;
| "
| double quote
|-
| & # 39 ;
| '
| single quote
|-
| & # 40 ;
| (
| left parenthesis
|-
| & # 41 ;
| )
| right parenthesis
|-
| & # 42 ;
| *
| asterisk
|-
| & # 60 ;
| <
| less than
|-
| & # 62 ;
| >
| greater than


and here is a reference page for more character codes:

http://www.w3.org/MarkUp/html-spec/html-spec_13.html



EDITED: For Chrome users, I just tried an extension called "Word Replacer," found here:

https://chrome.google.com/webstore/detail/word-replacer/djakfbefalbkkdgnhkkdiihelkjdpbfh?hl=enhttps://chrome.google.com/webstore/detail/word-replacer/djakfbefalbkkdgnhkkdiihelkjdpbfh?hl=en.

It's not as easy to access as FoxReplace, because you can't add it to the toolbar. To configure it, click the menu button (the three stacked bars), select More Tools, then Extensions, then find Word Replacer in the list, and click on the Options link. Based on a very limited test, it seems to work fine.

One difference seems to be that Word Replacer runs automatically on every webpage you go to. In FoxReplace, I believe you can do that, or you can simply click the toolbar button to run it on demand. (I think you can also set up FoxReplace to work only in certain domains, although that's more than I cared to fuss with it.) I don't know what effect Word Replacer's lack of control would have on normal web browsing - an option might be to keep it disabled most of the time, and only enable it when you need it.
 
Last edited:

pocossin

visitor
Joined
Feb 7, 1970
Messages
4,521
Reaction score
181
Would you give a specific example of this garbling? I almost never consider the archives when considering a query in Shared Reading. Wouldn't it be better to consider what is actually presented and to ignore the archives?
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
Here's one:

http://www.onlineclarity.co.uk/friends/showthread.php?329-Lots-of-52

You'll see all through it strings that look like "& # 34 ;" without the spaces. They're actually supposed to be characters like apostrophes and greater-than symbols, but the forum software isn't rendering them properly (I think it has to do with versions and upgrades and so forth). It makes it difficult to read the page.

I come across this mostly when doing hexagram searches, and other searches as well, since - fortunately - the search features look in the archives :).

These add-ons are just to make things easier to read.
 

pocossin

visitor
Joined
Feb 7, 1970
Messages
4,521
Reaction score
181
Thank you. I now see the difficulty that you aim to correct.
 
G

goddessliss

Guest
I'm with Lisa - I find them difficult to read so now when I come across them I don't even bother trying - seems such a waste of what could possibly be very helpful to the querant/member.
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
Liss, what browser do you use?
 

pocossin

visitor
Joined
Feb 7, 1970
Messages
4,521
Reaction score
181
If someone were to complain to Hilary, she might get the vBulletin people to correct this.
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
Mayyyybe, but I suspect it's more likely Hilary would spend several times longer trying to get their attention than they would trying to fix it.
 

Trojina

Supporter
Clarity Supporter
Joined
May 29, 2006
Messages
26,920
Reaction score
4,424
I agree it seems a waste at times not being able to read old threads but I don't think much can be done because I think we have asked before. So maybe your helpful solution, for the computer savvy, is the way Lisa. Hilary might not see this thread anyway so I'll point her at it, then if she's too busy now she might look later.
 

hilary

Administrator
Joined
Apr 8, 1970
Messages
19,148
Reaction score
3,418
It happened with the import from Discus (anyone remember that? ;) ) to Vbulletin. Possibly some clever MySql-database-expert could, for a suitable fee, run a find-and-replace that would put it all to rights - or corrupt the entire database beyond repair, of course. First find your expert...
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
You don't have to be all that computer savvy to use an add-on, though, I don't think. The one for Firefox is pretty simple, if you pay attention to just what you need, and ignore all the other confusing stuff. (That might be easy for me to say, now that I've figured it out, lol.)

I think I remember from other threads that you use Internet Explorer, though, Trojina? I'm not even sure if I.E. has add-ons. I mostly know about this Firefox one because it's the one which applies to meeeeee, and I figured I may as well let other Firefox users know about it. I haven't spent much time researching options for other browsers. May or may not do so. (Maybe I should, now that I've started something :eek:uch:...)

If you haven't yet, maybe don't make a fuss to Hilary. There's most likely nothing she can do, and the point of this was really just to let Firefox users know about something helpful. (I did not think this through enough.)

[Cross-postinggggg....thank you, Hilary. The add-ons do exactly the find-and-replace you're talking about[SUP]*[/SUP], pretty painlessly. I was tickled to find it, and it's absolutely all I need or want.]



[Edited] [SUP]*[/SUP]without messing with the actual database, so there's no chance of corrupting anything. So - even better!
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
Apparently I.E. does support add-ons, because there is a "Manage add-ons" item in the I.E. Tools menu.

Whether there is one for find-and-replace on webpages, though...haven't found one yet. I did find an article someone wrote about the topic last fall, who said they don't know of one for I.E.:

'How to "Find & Replace" Text Directly in Your Web Browser'

http://digiwonk.wonderhowto.com/how-to/find-replace-text-directly-your-web-browser-0157379/

Unfortunately, I've been unable to find any such extensions for Safari, Opera, or Internet Explorer. The latter had the functionality baked into older versions, but the current IE builds do not offer a search-and-replace function.

So maybe if you're using an old version? (Which is probably not recommended, though, for malware-security reasons, or so they say.)
 
P

peterg

Guest
Another solution for IE users.

Many of the older threads in the archives are quite hard to read because of the garbling.

I have a HP smart print add-on on Internet Explorer. I have it on Chrome too.
I prefer to use IE because I find it hard to find complete web page address on Chrome whereas on IE you just click on properties.
The problem is IE doesn't have the same range of add-on support.
But there is another solution apart from add-ons. It might sound complicated but its fairly easy to use.

Open Reply to thread and Go Advanced. If the thread is closed to replies, open the reply page on any thread in a New window.
Drag the garbled text to the reply page. (there is a limit of 20,000 characters.If the thread is less than 20K char. right click and select all).

Click top left a/A (Switch editor to WYSIWYG mode).{ I don't know what that means}.
Click second tab top left (remove format).
Click a/A again.(Switch editor..) That should get rid of the garbling.Temporarily.
Click Preview to read the full version thread. (the preview limit is 20,000 characters).

I just tried this on 'Lots of 52'. And another thread from the archives. Works fine.

Problems-risks:
I just discovered this solution so I'm not familiar with it.
I don't know what those a/A buttons normally do.
You might inadvertently submit the text as a post, so you need to be watchful. You can also cancel the text on the Quick Reply page.
 
Last edited:

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
Thank you, Peterg. :)

I just tried your method, and I see that, yes, it does magically get rid of the garbled characters. Very creative use of two editing buttons I'd never noticed or used. (I think I'll actually play with them to see what they do in a normal editing situation :D.)

Two problems:

1. I did run into the 20,000 character limit with the "Lots of 52" thread - I had to chop a fair amount of it off or else I only got an error message.

2. You end up with an upside-down thread, with post #1 at the bottom.

It's easy enough to read from bottom to top; the character limit may be a bigger problem. Maybe there's some work-around...
 
P

peterg

Guest
If you use the Preview for ''Lots of 52'' you will be told 'The text you have entered is too big (33150 characters) The number varies each time but its about that number.
So you can scroll down the page about half way and drag that much.

Alternatively you can skip the preview.
Drag the lot to the quick reply page (somehow it fits in) and Reformat there. Then go to Advanced page. You should see the reformatted text from the top of the page, newest post last.
The information above each post (date member location..) displays oddly, vertically that is.Hadn't noticed before. Maybe the preview is a better view.

I'ts still a bit new and trial and error but clicking on the editing tabs get's a result and you need to wait a couple of seconds after each click.

Hope we don't break the Forum.
 
Last edited:

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
Well...unless I'm missing something, I think the only danger is as you said - posting a "reply" you didn't intend. And if that happens, it's simple enough to delete it.

(That is, if people notice they've posted an accidental reply. That might be a problem. I've seen double posts on the forum from time to time, and clearly the person posting didn't know they'd done that.)
 
P

peterg

Guest
I prefer IE because it's easier there to find the complete web page address, although generally Chrome is probably more secure and superior.

I already have Chrome on the taskbar and just tried Word Replacer. It takes just seconds to install-enable WR and its simple to use.
Using Lisa's lists I added 5 substitutions in Word Replacer and that got rid of all the garbling in the thread ''Lots of 52''. http://www.onlineclarity.co.uk/friends/showthread.php?329-Lots-of-52

I opened an old Yilin thread linked in cjgait's ''Forest of Changes Translation'' thread and added three more substitutions and that got rid of all the garbles there.
Except for &uuml: in post#22. I couldn't figure that one out.
http://www.onlineclarity.co.uk/frie...st-of-Changes-Translation&p=208620#post208620

So it looks like Lisa hit the nail on the head first try. The solution is Word Replacer.
If you havent got Chrome, install it.Or Firefox ?

Like Lisa I've never noticed or used those AA edit tabs before and it is interesting playing around with them.
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
I'm glad it works :)

Yes, I looked at the ü and I don't know what it is, either. Hopefully it doesn't show up very often, or maybe someday we'll figure it out.

EDITED: Found ü (and a lot of other codes) at the following webpage - it's an umlaut. It works to just copy (ctrl-c) the umlauted character directly from the screen and paste it into the substitution list, at least in FoxReplace.

http://www.thesauruslex.com/typo/eng/enghtml.htm

I prefer IE because it's easier there to find the complete web page address

You mean the url that shows up in the address bar isn't the complete web address? I'm probably missing something obvious...

I like IE's ability to save in .mht flat-file format. Why all browsers can't do something similar, I have never understood. But there are a couple Firefox add-ons I rely on, so I stay with Firefox.
 
P

peterg

Guest
http://www.onlineclarity.co.uk/frie...i-Lin-(-Aring-Otilde-Ntilde-)&highlight=yilin
That's the direct link to the old Yilin thread.
Any idea what's in the first post: & Aring; & Otilde; & Ntilde; And in the thread title garble.
Edit : I see them in the link above on HTML encoding of foreign language characters.Its the code for (?Å???Õ?Ñ). Probably Chinese characters displaying as question marks and letters.
A lot more HTML codes here http://www.rapidtables.com/web/html/html-codes.htm

Another cool list. & uuml ; is ú I think.Its a name.Lu Shang.

Chrome: I mean that I find it hard to set up links to Chrome pages, for some reason.Where's the address page.

Must check out Firefox sometime, and FoxReplace.
Best Browsers http://gadgets.ndtv.com/apps/features/the-best-free-web-browsers-for-windows-656453
Good review but if you're on a limited data plan watch out for the advertisement on this page.

No idea what IE flat-file format is.

A couple of years ago I made a passing reference to garbling in old threads, and wondered if it was fixable. No one else seemed to notice it. So now I have the answer.
 
Last edited:

Hujambo

visitor
Joined
Jun 26, 2015
Messages
171
Reaction score
12
It happened with the import from Discus (anyone remember that? ;) ) to Vbulletin. Possibly some clever MySql-database-expert could, for a suitable fee, run a find-and-replace that would put it all to rights - or corrupt the entire database beyond repair, of course. First find your expert...

I might be able to get a something together for you to sort this issue out. No promises, and there will be a bit of a delay whilst I organise access to a test environment. I should probably find out what Vbulletin is first, LOL, but I already know it will be a something-or-other front-end to a MySQL back-end so it's just a case of discovering the field and table names they are using within MySQL. If I can find a way to get it done simply I'll let you know. In relation to corrupting the entire database, I would never advocate running this kind of fix on live data if it's in any way avoidable. So I'm fairly risk-averse when it comes to remediating data.

By way of background, I usually solve these types of problems for a living, ie. last year I was resolving the issue of repeated server crashes due to bad data (although on a far larger scale and they were a different type of databases). That was at one of the largest technology environments in Australia. Someone did corrupt an entire mission-critical database trying to fix it, but I put it back together. I hate it when IT people create a mess for their clients.

Cheers,
Robyn
 

Hujambo

visitor
Joined
Jun 26, 2015
Messages
171
Reaction score
12
Checked out the vBulletin forums, the issue is not uncommon and yeah, it's definitely the fault of whoever did the import. They should have aligned the character sets before importing the data. I can immediately see at least one way this issue might be resolved, but I will continue investigating to see exactly what the options are. You want the simplest, most reliable and least risky option.
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
Hi Peterg, I forgot to reply to this and was reminded by the appearance of newer comments.

Any idea what's in the first post: & Aring; & Otilde; & Ntilde; And in the thread title garble.
Edit : I see them in the link above on HTML encoding of foreign language characters.Its the code for (?Å???Õ?Ñ). Probably Chinese characters displaying as question marks and letters.

Yes, all I could find in Google is those capital letters, and I don't know what they mean either.

I've also noticed on some pages apostrophes are turned into question marks (e.g. don't is turned into don?t. You can't replace them, though, because you'd also replace all the real question marks.

So, substitution add-ons are not perfect. But they seem to fix a high percentage of the problem, which is good enough for me.

& uuml ; is ú I think.Its a name.Lu Shang.

Yes, a "u". (And I also don't care about the accent, just that it's a "u" instead of an "& umml ;")

Chrome: I mean that I find it hard to set up links to Chrome pages, for some reason.Where's the address page.

The url in the address bar isn't the same in all browsers? I didn't even know that.

No idea what IE flat-file format is.

File > Save as... > Save as type: Web Archive, single file (*.mht)

I used this all the time when I used IE. Instead of saving a web page in pieces (a file plus a folder full of other files) it was just name-of-webpage.mht. One file.

A couple of years ago I made a passing reference to garbling in old threads, and wondered if it was fixable. No one else seemed to notice it. So now I have the answer.

These add-ons may not have existed a couple years ago. (And they may not exist a couple years from now, given the nature of 3rd-party addons and software in general...)
 

hilary

Administrator
Joined
Apr 8, 1970
Messages
19,148
Reaction score
3,418
Checked out the vBulletin forums, the issue is not uncommon and yeah, it's definitely the fault of whoever did the import. They should have aligned the character sets before importing the data. I can immediately see at least one way this issue might be resolved, but I will continue investigating to see exactly what the options are. You want the simplest, most reliable and least risky option.

Thank you so much for looking into this! It did occur to me it might be something that could be solved with find-and-replace in phpMyadmin. But I know zilch about mySql and am basically too scared of breaking things to click anything in there.
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
[Edited: Oh good, for once I'm glad to see cross-posting - Hilary already saw your comment, Robyn.]


Hi Robyn,

Ah, someone who knows what the words mean and what the problems are! :)

BUT...

...this sounds like work, and time. The add-ons are almost no work, and they take very little time. They seem to solve the lion's share of the problem, and they pose zero risk to the forum, not even a small one.

And this thread isn't the right place or audience. You'll need to talk with Hilary about it. Her email is down under Contact at the bottom of the page: hilary@onlineclarity.co.uk, or you could send her a private message through the forum.
 

Hujambo

visitor
Joined
Jun 26, 2015
Messages
171
Reaction score
12
No worries, Lisa. I'm just curious by nature and it was an interesting challenge. Happy to leave you all with the solution you already have.

Cheers
 

Hujambo

visitor
Joined
Jun 26, 2015
Messages
171
Reaction score
12
Thank you so much for looking into this! It did occur to me it might be something that could be solved with find-and-replace in phpMyadmin. But I know zilch about mySql and am basically too scared of breaking things to click anything in there.

Exactly Hilary, find-and-replace in phpMyadmin is the simple way to go. A simple way to test is to put extra parameters/criteria into the find so that you only return one record in the search result, such as a post title along with one of the special characters you want to replace. If it does something unexpected, you have only "mooshed" one record, which I am sure you can just go back and edit to fix up if necessary. Obviously take a backup of the MySql database beforehand.

With these things, the more preparation and testing you do, the lower the risk, but you can never eliminate the risk entirely.

One of my closest friends has a web hosting/development company and works with phpMyAdmin every day. I will ask him if he is aware of any pitfalls with the find-and-replace that I cannot see and send you an email if you like.
 

Hujambo

visitor
Joined
Jun 26, 2015
Messages
171
Reaction score
12
And this thread isn't the right place or audience. You'll need to talk with Hilary about it. Her email is down under Contact at the bottom of the page: hilary@onlineclarity.co.uk, or you could send her a private message through the forum.

My sincere apologies for posting in the wrong place. I am very new and this is where the issue was raised, so it's an easy mistake. It won't happen again.
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
Yes, it is where the issue was raised, and you didn't do anything wrong. I was nervous about starting more of a fuss than I ever intended, and I wanted to be clear that I wasn't asking for a fuss to be made (still am not, though of course it would be nice if the archives were garble-free. It's a fine line I'm treading here.)
 

Hujambo

visitor
Joined
Jun 26, 2015
Messages
171
Reaction score
12
Aaah, I get where you're coming from now. :) It's not a hassle at this stage, just exploring the possibility and seeing how difficult/easy it might be to resolve. What is really cool is that the interaction has this feeling of "flowing" nicely at this time. Hilary knew exactly what I had in mind by referring to the simple solution. I thought it would be tricky to gain an understanding of vBulletin and it turns out be just another flavour of something I understand, plus it has good support (if you can do the geek-speak that is). If it was ever going to get resolved permanently, now is the time to make good progress. The stream is flowing our way - so to speak. If it gets too hard, time-consuming or costly - we already have available this excellent workaround that you have all been working on, by using the browser tools.
 

Liselle

Supporter
Clarity Supporter
Joined
Sep 20, 1970
Messages
12,849
Reaction score
2,387
I'm happy to hear all that, Robyn. :) (This "flowing" you speak of - is it for sale somewhere? I could use some! :D Hahaha)

One thing I noticed in using the browser add-on:

I've also noticed on some pages apostrophes are turned into question marks (e.g. don't is turned into don?t. You can't replace them, though, because you'd also replace all the real question marks.

My concern is not so much that there might be straggler garbles (a few stragglers can easily be ignored when most of them are gone) - but why it happens and if there are further implications.

For example, see this thread:

http://www.onlineclarity.co.uk/friends/showthread.php?31-Multiple-moving-lines-a-dramatic-view

In post #2, an apostrophe in the word "haven't" is displayed as & # 3 9 ;. But in post #3 (second-to-last paragraph), the apostrophes are replaced by single question marks.

Why would that happen? As far as I can tell, the edit box used for writing posts doesn't allow more than one thing to happen when someone presses the apostrophe key. It's not like Microsoft Word, where you can selectively choose straight or curly apostrophes and quote marks at different places in a document.

I don't know if there are other such oddities.

Also, Hilary's blog seems to have its own set of (different?) garbles. There are a lot fewer of them, and I can't find an example now when I want one. I think the blog might come from a different place than vBulletin (?), so it may have nothing to do with what you're investigating.

(P.S. The Yi part of your brain will love Hilary's blog, if you haven't already discovered it.)
 

Clarity,
Office 17622,
PO Box 6945,
London.
W1A 6US
United Kingdom

Phone/ Voicemail:
+44 (0)20 3287 3053 (UK)
+1 (561) 459-4758 (US).

Top