Showing posts with label OwnCloud. Show all posts
Showing posts with label OwnCloud. Show all posts

Tuesday, August 23, 2022

OwnCloud Manual import of data

If you are here, you've obviously started to use Owncloud and found that importing the mountain of data you want accessible is painful. So what if you have gigabytes of files to add to your ownCloud server? You can of course use the desktop client to sync all those files, but what if we were talking about almost a terabyte?

Add files manually to your ownCloud server

This is probably the quickest and less complicated way of adding files. There are many alternative ways to transfer your files to your ownCloud server:

      • FTP Upload;
      • SCP Upload;
      • Local disk to ownCloud disk transfer;
      • USB disk to ownCloud disk transfer.

It's just as simple as dragging a folder to another folder. The only problem is that you might not know yet where ownCloud files are stored. I made life a little bit easier with this nifty little Bash snippet that automatically finds your ownCloud data directory:

find / -name config.php 2>/dev/null | while read line ; do cat "$line" | grep " 'datadirectory'" ; done

Your output should then look something like this:

ownCloud Data Directory Snippet

So according to the snippet of code my ownCloud data directory is located at /var/www/owncloud/data so let's cd into that directory to check out what's in there.

ownCloud Data Directory Contents

As you can see, we have multiple files in here. My ownCloud username is bart, so let's take a look in that folder.

ownCloud Data Directory Closer Look

Now this looks interesting. If I check my ownCloud web portal I can see that the directory structure is the same:

ownCloud Default Web Interface Folders

So I created a folder called TestFolder123 in the web interface and executed ls -la on the server's console.

ownCloud Default Web Interface With Test Folder

ownCloud Server Console Test Folder

So I tried to do the same thing, but vice versa. I created a folder on the console side, updated the folder's ownership and afterwards checked if the folder appeared in the web interface, with some surprising results:

ownCloud Server Console Test Folder 2

ownCloud Default Web Interface Without Console Test Folder

What the heck?!

Where did my folder go??!

Okay.. let me explain what's going on here. The ownCloud system basically expects two things from a file/folder in order to be indexed as a file:

      • The file/folder needs to be physically available;
      • The file/folder needs to exist in the ownCloud database.

Moving a file/folder into ownCloud's data directory doesn't add it to the database automatically. Luckily, the ownCloud developers were kind enough to create a command line tool called occ that can manually add missing files and folders to the ownCloud database. You can run the following snippet on your server to check where this utility is located on the local file system:

find / -name "occ" 2>/dev/null

Which outputs the following on my demo server:

OCC Binary Location

All you need to do is cd into that folder (in my case /var/www/owncloud) and run the following snippet:

sudo -u www-data php occ files:scan --all

Where www-data is the user your web server runs on.

    Another option would be a command that scans a specific folder or user.  In this example, the username is "bart" and the folder to scan is  in the "files" directory called "temp2" 

 

NOTE: the path is not the full path in the file system but must start with the username.  For example my installation & data directory is located in /var/www/owncloud/data   all the user data is located in a folders (named by username), followed by a "files" folder and then any folders/directories created there after for the given user.

If you try to use a fully qualified path afer --path directive,  you will encounter the error

Unknown user 1 var

sudo -u www-data php occ files:scan --path bart/files/temp2

Further more, if you wanted to scan all the files and directories for a given user, you could run the following, where "bart" is the ownCloud username.

sudo -u www-data php occ files:scan bart

 And now, it is time for the results:

OCC Output From The Console

Positive Results In The Web Interface

YEAH!! So this command has definitely helped. Now you can add all your files directly into your ownCloud data folder without having to sync from the desktop client.


Wednesday, December 27, 2017

LVM Resize - Debian Virtual Machine

I recently setup an installation of Owncloud on a Debian virtual machine so that my family members could more easily share some holiday pictures.  Within a few weeks the usage increased substantially and I soon found that the VM was nearing capacity.  I had purposely utilized LVM just in case resizing became necessary.
There are multiple ways to accomplish this task and this is but one way that works for me and the Debian OS.  I opted to add a new (second) virtual disk and then expand the LVM, but it is possible to increase the original disk size and then follow similar steps.  While the utilities are the same, the process is different.  Please take note.
Critical Note: A mistake with the following commands could render your Virtual Machine unusable.  I take careful precautions when doing maintenance of this nature.  Due to this system being a VM, I was able to make a clone prior to making any changes.  A snap shot or copying the VM would also provide some protection.
My Victual Machine is running on OSX and VMware Fusion.  The Fusion software varies slightly from other VMware products.  I had to shutdown my VM to add the hard drive but I know in other VMware variants, that I've been able to add the disk on the fly.

Identifying the partition type

Confirm that the partition type is actually Linux LVM.
fdisk -l
fdisk of newly added disk
 /dev/sda5 is listed as Linux LVM and it has the ID of 8e. The 8e hex code shows that it is a Linux LVM, while 83 shows a Linux native partition. 
Below is the disk information showing that our initial setup only has the one 20gb disk currently, which is under the logical volume named /dev/mapper/Mega-root – this is what we will be expanding with the new disk.
Disk free newly added disk
Note that /dev/mapper/Mega-root is the volume made up from /dev/sda5 currently – this is what we will be expanding.

Adding a new virtual hard disk

First add a new disk to the virtual machine. Be sure to match the drive type.  Create the size disk that is needed for the increased capacity.  
Detect the new disk space
In my test for this example, as soon as I added the additional disk in through VMware it displayed through “fdisk -l” for me, you can see the second disk labelled /dev/sdb 
fdisk of newly added disk
If your system doesn't read the disk, you can run the following command to rescan.  Change host0 depending on your setup.
echo "- - -" > /sys/class/scsi_host/host0/scan
Reboot if needed...Ultimately a reboot will make the partition visible.

Partition the new disk

We now need to partition the new /dev/sdb disk so that it can be used, this is done by using fdisk.
fdisk /dev/sdb
This should provide us with the below prompt, the inputs I have entered in are shown in bold.
‘n’ was selected for adding a new partition.
root@Mega:~# fdisk /dev/sdb
Command (m for help): n
‘p’ is then selected as we are making a primary partition.
Command action
   e   extended
   p   primary partition (1-4)
p
As this is a new disk, we do not yet have any partitions on it so we will use partition 1 here.
Partition number (1-4): 1
Next we press the enter key twice, as by default the first and last cylinders of the unallocated space should be correct.
First cylinder (1-2610, default 1): "enter"
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-2610, default 2610): "enter"
Using default value 2610
‘t’ is selected to change to a partitions system ID, in this case we change to ’1′ automatically as this is currently our only partition.
Command (m for help): t
Selected partition 1
The hex code ’8e’ was entered as this is the code for a Linux LVM which is what we want this partition to be, as we will be joining it with the original Linux LVM which is currently using /dev/sda5.
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)
‘w’ is used to write the table to disk and exit, all changes that have been done will be saved and then you will be exited from fdisk.
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
By using “fdisk -l” now you will be able to see that /dev/sdb1 is listed, this is the new partition created on our newly added /dev/sdb disk and it is currently using all 20gb of space.
fdisk after partition created

Increasing the logical volume

Next we will use the pvcreate command to create a physical volume for later use by the LVM. In this case the physical volume will be our new /dev/sdb1 partition.
root@Mega:~# pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created
Now we need to confirm the name of the current volume group using the vgdisplay command. The name will vary depending on your setup, for me it is the name of my test server. vgdisplay provides plenty of information on the volume group, I have only shown the name and the current size of it for this example.
root@Mega:~# vgdisplay
  --- Volume group ---
  VG Name               Mega
  VG Size               19.76 GiB
Now using the vgextend command, we extend the ‘Mega’ volume group by adding in the physical volume of /dev/sdb1 which we created using the pvcreate command just before.
root@Mega:~# vgextend Mega /dev/sdb1
  Volume group "Mega" successfully extended
Using the pvscan command we scan all disks for physical volumes, this should confirm the original /dev/sda5 partition and the newly created physical volume /dev/sdb1
root@Mega:~# pvscan
  PV /dev/sda5   VG Mega   lvm2 [19.76 GiB / 0    free]
  PV /dev/sdb1   VG Mega   lvm2 [19.99 GiB / 19.99 GiB free]
  Total: 2 [39.75 GiB] / in use: 2 [39.75 GiB] / in no VG: 0 [0   ]
Next we need to increase the logical volume with the lvextend command (rather than the physical volume which we have already done). This means we will be taking our original logical volume and extending it over our new disk/partition/physical volume of /dev/sdb1.
Firstly confirm the name of the logical volume using lvdisplay. The name will vary depending on your setup.
root@Mega:~# lvdisplay
  --- Logical volume ---
  LV Path                /dev/Mega/root
  LV Size                18.91 GiB
The logical volume is then extended using the lvextend command. We are extending the original logical volume of /dev/Mega/root over the newer /dev/sdb1
root@Mega:~# lvextend /dev/Mega/root /dev/sdb1
  Extending logical volume root to 38.90 GiB
  Logical volume root successfully resized
If you like you can then run vgdisplay and lvdisplay again to confirm the size of the volume group and logical volume respectively, I have done this and I now have the following.
  LV Size                38.90 GiB
  VG Size                39.75 GiB
However if you run a “df” command to see available disk space it will not have changed yet as there is one final step, we need to resize the file system using the resize2fs command in order to make use of this space.
root@Mega:~# resize2fs /dev/Mega/root
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/Mega/root is mounted on /; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 3
Performing an on-line resize of /dev/Mega/root to 10196992 (4k) blocks.
The filesystem on /dev/Mega/root is now 10196992 blocks long.
Alternatively if you’re running the XFS file system (default as of RedHat/CentOS 7) you can grow the file system with “xfs_growfs /dev/Mega/root”.
Rather than resizing the file system manually, you could instead use the -r option of the lvextend command which will automatically resize the file system to make use of the additional disk space.
The resize took a minute or so to complete (it will depend on the disk speed and size), running the “df” command now shows the correct disk space for /dev/mapper/Mega-root
Disk free on expanded LVM

Sunday, August 16, 2015

Gmail SMTP & OwnCloud

While trying to configure gmail SMTP on my OwnCloud installation, I continued to run into an authentication issue.  Oddly enough, I have several other areas where I have this working without issue (server & webcam) and so I'm no stranger to the configuration and setup.  I was nearly stumped when I found a solution that may help others as well.

I finally found the following "https://accounts.google.com/UnlockCaptcha".  Captcha was causing the hiccup.  By unlocking the account, all was back to normal and emails could be sent from my cloud server. 

My OwnCloud 8.1.1 config is as follows;

  'mail_smtpmode' => 'smtp',
  'mail_from_address' => 'mygmailacct',
  'mail_domain' => 'gmail.com',
  'mail_smtpsecure' => 'tls',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'smtp.gmail.com',
  'mail_smtpport' => '587',
  'mail_smtpname' => 'mygmailacct@gmail.com',

  'mail_smtppassword' => 'MyPassword',