Question Corrupted files

[davidm]

I have a large directory of Flac files on my DS418Play. The storage pool is 3 x 12Tb using Raid 5 and in addition this directory is backed up using Hyper Backup. I noticed some of the original files were corrupted and not being recognised as flac: the file name had been changed to rubbish, the file suffix was missing and as a result MacOS thought they were Unix Executable files. I discovered that if I renamed each as a .flac file it returned to being recognised by MacOS as flac and played ok.

To solve the name change I went to Hyper Backup, found the copy (which has the correct full name and flac suffix) and copied it back. But the copied file also becomes corrupted and ends up exactly the same as the original!

Why is it happening? How do I stop it? How can I copy the (apparently correct) file back without it also becoming corrupted?

 

[davidm]

If instead of copying from backup I instead download to my Mac SSD the copied file is perfectly OK. This is clearly a Synology problem....

 

[jeyare]

nothing happens just by itself.
when you will decide, as you stated: “This is clearly a Synology problem....”
you will never discover the exact problem source.
Reason:
- file suffix is part of the filename, there is also binary signature in the file. then a destroying of the suffixes is possible during incorrect data handling between data operation host (computer) and storage host (NAS). There is also another important point data has been corrupted during different “parallel” event.
- I found in your previous post from August 2019, that you accidentally destroyed your data without backup of them and then recovery has been processed with “solved” status
- then you have first source of thinking, why you have some of them in the mentioned stage of missing suffix
- you wrote, that you have now also HB. But when you have backup of corrupted original, you will get just same during the restore, no matter of the suffix.

Proposed test scenarios:
Try to open your HB portal and explore the file, that was corrupted in your Shared folder. Then copy of the file somewhere (to your computer, to your NAS, ... both of targets). Is it ok? Is the file recognized as valid flac file? With valid flac ext?
Then same in same situation - restore just the file from the backup to original Shared folder. Check it also.
Send here a screenshots from the tests (corrupted file, HB portal, file extraction, restore).

 

[davidm]

I may have made a mistake on my previous post but that does NOT mean I am making a mistake now...

• Sample folder on DS418Play with damaged files (screenshot enclosed)
• My first backup (also to DS418Play) using Hyper Backup is from 14/4/2020. Screenshot of that same backed up directory enclosed - all files are shown as flac in the filename.
• I have tried copying all of the back up copies of the damaged files in that sample directory to the multimedia directory of DS418Play (screenshot enclosed) - none of the files are written as flac. Copying an apparently good backup on DS418Play to another location on the DS418Play does not seem to work.
• I have tried downloading 1 damaged file from backup to the main data folder of DS418Play (screenshot enclosed) - it is not written as flac. I repeat - copying an apparently good backup on DS418Play to another location on the DS418Play does not seem to work.
• I have tried downloading a backup copy of one of the damaged files on DS418Play to Mac SSD (screenshot enclosed) - it IS written as flac. I conclude therefore that the backed up file is NOT corrupted. And also that the original file was also not corrupted when backed up!
• I have also previously tried downloading from backup on DS418Play to my SSD and then copying back to the DS418Play - downloading a single file and copying back to DS418Play works, but multiple file copy (download tracks 5-8 individually and then copy them all together) results in damaged files.
• I have many directories with this problem - something is corrupting the files. Something is also corrupting good backup files that are copied or downloaded.

Thank you.

 

[jeyare]

9 flac files in your Mendelssohn folder
you don't have issue with files 1/2/3/4/9
files 5/6/7/8 contains in file name probably two quotation marks (sorry, low resolution is there) what isn't contain in files 1/2/3/4/9

evaluation:
- follow official Syno KB - quotation marks are one of the invalid characters in file/folder names
- characters \ * ? " > | # % contained in the filename will be replaced by - after downloading.

reason:
SMB prerequisites

an action:
you need search all possible incorrect filenames and rename them, or use script in CLI

verdict:
this isn't Synology problem, as you can see

Btw:
- also Shared folder names have characters policy - Names cannot include the following characters: ! " # $ % & ' ( ) * + , / : ; <= > ? @ [ ] \ ^ ` { } | ~

 

[davidm]

Your explanation fails to answer some important questions:

• How could a filename with a character that is not permitted (") have been created on the NAS and then successfully backed up (Screenshot 1)?
• How can that perfectly good backup file become corrupted when copied by Hyper Backup (screenshot 2)?
• How can that same perfectly good backup file be downloaded (with " changed to _) by Hyper Backup to my SSD (Screenshot 3) and NOT become corrupted?
• How is it possible to copy that file on the SDD back to the NAS (Screenshot 4) and then rename it (_ changed to ") so that I have a filename (Screenshot 5) you say is not permitted?

It is not illegal characters in filenames that is corrupting these files (and folders) - it is the NAS operating system or Hyper Backup.....

 

[jeyare]

look, more important question is: If you are inventor or just claimer?
instead of crying, you could have done a set of tests till now. As I can see. No test, no results.

1. as you can see in my reason - it’s about SMB = Server Message Block protocol or just “samba“ is a network protocol used by Synology and rest of the world OS producers for a sharing of files and another useful services.

2. SMB isn’t product of Synology

3. in DSM/ File station you can create filename include quotation marks. Tested by me yesterday before I wrote my last post. Then here is still valid what I wrote.

4. Hyper Backup create just backup batch in case when you don’t use rsync (over SMB). Then you transfer the batch and no files over SMB

Final stage is about your setup of DSM/File services and your used process of data transferring between NAS and your computer:
- extraction from the Hyper Backup to somewhere by SMB will cause this problem
- AFP has different scenario

Then, if you like be wiser just be helpful. Otherwise no one has a motivation to help you.

PS: there is a way how to repair in smarter way, only when you don’t use Windows OS connected computers. SSH to NAS, find smb.conf, then write (or rewrite) these lines:
mangled names = no
dos charset = ISO8859-1
unix charset = ISO8859-1
display charset = ISO8859-1

 

[davidm]

I think the screenshots show that i have been testing! Are you for real? Employed by Synology?

 

[jeyare]

the samba.conf needs to be rewritten every time after DSM update

Samba “mangle” the original names, because it’s based on characters set rule of inventor of such protocol = Microsoft

 

[davidm]

your answers may be clear to someone who doesn't need technical help but they are not at all clear to someone who does need help!
Simple question, simple answer please - if I repair the damaged files (the way I know how) and avoid using files or folders with those characters can I be sure this problem will not return?

 

[jeyare]

My second post:

an action:
you need search all possible incorrect filenames and rename them, or use script in CLI

is there something not clear?

 

[DeltaGolf]

I'm pretty sure you can do this using 'find and replace' in Finder.

@davidm Select all the files with (say) # in their name. Control-click to get the fly-out menu. Choose "Rename x items...". Chosse "replace text", put # in the 'Find' and whatever you want in 'Replace with'. I'd suggest no as short for number if that's what it means. But don't use no. with a dot!

Hope that helps!

 

[fredbert]

iTunes defers some characters to underscore to avoid filesystem file name issues. I've noticed (can't remember which) one media package on DSM will ignore files with '?' so have had to rename the few that have this.

 

[jeyare]

@DeltaGolf
one of reason of this forum is to keep all information provided for all readers, not just for the OP. Because each of readers can find a way from the recommendations.
From some of readers it is necessary to be written in easy language. What is absolutely OK.
From some of readers is useful to get also a deepest technology background. Because a curiosity is a basic essence of knowledge.
I'm one of the person who try to get people in this forum to communicate decently and to support the basic principles of helping people to people = free of charge, without support from Vendor. Just as group of enthusiast with this kind of willingness.
Some person sometime use different attitude and it is up to moderators to keep mentioned environment. Sometimes it is enough to warn people about inappropriate behavior, sometimes we literally kick them out. Everything for your feel good here = to find a help make your systems/fun/environment better.

-----

So back to this kind of situation (not first time seen here in the forum).

My common approach is based on analyze of the problem instead of shooting into dark (common attitude of impatient).
My first post (#3) was about this attitude -

nothing happens just by itself, when you will decide, as you stated: “This is clearly a Synology problem....” you will never discover the exact problem source

There was proposed set of analytic steps for the member and request for the screenshots
Also I found, in forum history, that this member doesn't have deep dive in basic Linux experiences
So I understood that he needs to get a simple answer.

Then my second post (#5) was strictly structural = findings/evaluation/reason/an action/verdict:
in the Action stage it was proposed by me really simple guide how to solve the problem:

you need search all possible incorrect filenames and rename them, or use script in CLI

but also for rest of readers some deep dive hints and explanations.

However, this member tries to refute (post #6) my recommendation, without a reasonable evidence, just with same repeated consequences, without cross tests of proposed solution, also in some kind of "nervous" attitude. Then he got my (post #7) question: if we will eat time with consequences or test now the proposed solution ?
All of us spend time here without the right to a reward. Then waste of our willingness to help is a waste of our time. What is an unacceptable loss to me. We have too little time in this world to spend it with a hassle.

Now:
- unskilled persons know, that they have to avoid use some characters in the Folder/File names
- skilled knows, how to use CLI to solve it in deeper level

What I don't like:
- an over-smart attitude, when from beginning is someone other responsible for his own troubles, even when he is not the skilled person
- people who lose touch with a humility
- fighting here who is smarter.

Final stage:
When someone here takes umbrage against me, based on my behavior = so take it as my excuse.

 

[jeyare]

in SSH:

find /volume1 -name "*[\+\{\;\"\\\=\?~\(\)\<\>\&\*\|\$]*" -ls -type f > /volume1/volume1-file_not-allowed-char.txt

when:
- volume1 is variable, then you can use volume1, volume2, volume3 .... up to your NAS setup

then you can get output into:
volume1-file_not-allowed-char.txt
it's structured text, then you can use it as import to CSV editor/XLS

tested, works

then RENAME in DSM/File Station is needed (avoid any complication in desktop computers with SMB)

I hope, that for the Rename of file in File Station is not necessary to describe.

 

[jeyare]

... you can use characters like ~ for example in Mac filenames. I do it all the time myself. But you can't on a Synology. Dunno why but it takes the huff and I've seen much the same for my folders which I add a ~ to. So best thing is rename the file to exclude things like #. I'm pretty sure you can do this using 'find and replace' in Finder. I'm also pretty sure I did something like this recently and found help on a Mac site, Tidbits maybe. There are apps but Finder is free!

This isn't correct advice, reason:
- best think isn't rename the files in Finder when you are connected to NAS by SMB. You have to use source system directly, then by DSM/File station or by SSH are one and only right approach and 100% sure
- each external APP connected into the NAS by SMB can fail in same way, because they will use the same SMB