• Home
  • Information Tech

Using Git and TKLDEV to build your own customized Turnkeylinux (etc) resource set

Learning anything takes some practice.  I have been hoverting around my reentry for a bit now and went in for a landing.  I am going to do my best to build a customized set of builds based upon the Turnkeylinux set of apps, but one which is more flexible and more customizable to my own needs.

Here are some of my process notes.  It seems to be going pretty well.

  1. Sign on to hub.turnkeylinux.com
    1. Set up a Micro Server via the hub to run TKLDEV
  2. Sign on to Github
    1. Go to your https://github.com/turnkeylinux-apps
      1. Choose the Core App
        1. Fork it into your own repository set (this will be suggested by Github)
      2. Choose the TKLDEV App
        1. Fork it into your repository set
      3. Choose someother App you want to work with  (I will do Joomla and Odoo for now)
        1. Fork it (or them separately) into your repository set.
    2. Go to your Repository Root
      1. For each new repository
        1. Make a new "Branch" with some operative name for your work
        2. Option - but recomended
          1. Make the new branch be the "default" branch for you (rather than "master" being default)
  3. Log into your TKLDEV Box via SSH - ie Putty.  Iin my case the TKLDEV generated on hub.turnkeylinux.com)
    1. cd /  (out of /root)
    2. cd turnkey  (1 and 2 together could be "cd /turnkey")
    3. cd products (1 and 2 and 3 together could be "cd /turnkey/products")
    4. Clone each of your repositories (using the -b default_branch_name) to your products directory using a command like the following
      1. git clone https://github.com/your-user-name/your-turnkeyapp-repository -b default_branch_name
      2. cd (from products) to one of the new cloned directories
      3. issue the "make" command
    5. If you want to remove and start over I use the following
      1. rm -rf directory_name
      2. I may have to "restart" the server if I cannot complete because some chroot is running after a "make command"..  After reboot again use the rm -rf directory_name command to clear it.
  4. I am right there now.  I will start working with Git on my personal branches and see what modifications I can run
  5. My specific focus will be on TKLDEV modifications.  
  6. If I want to use those modifications actually "inside" my operating TKLDEV appliance  I will likley need to Link the Code directly - which means this is a Line of my TODO list.
  • Now to BuildTasks
  • Forked https://github.com/turnkeylinux/buildtasks into my repository set -ie https://github.com/my-user-name
    • Created a Branch - my name
    • Made the Branch "Default"
  • Opened GitHub Desktop
  • Cloned https://github.com/my-user-name/buildtasks into my GitHub Desktop
    • copied bt-xen into bt-mine
    • copied /patch/xen/ into /patch/mine  (min is a pseudonym)
    • Made a Commit with Summary and Detail Annotation
    • Pushed the Commit with Summary and Detail Annotation
  • Opened my TKLDEV Machine with PUTTY (SSH)
    • Navigate with
      • cd /turnkey
    • dir (to be sure)
    • rm -rf buildtasks
      • that removed the buildtasks directory so that I can clone my repository
    • git clone https://github.com/my-user-name/buildtasks -b my-branch-name
      • technically the branch part should be covered automatically as I set it as default in github directly, but I try to be safe and specify the branches explicity.

That takes me to where I am now.

I need to start working on my Build Processes next.  Also, quite explicitly, I will need to overtly change the "include" command for buildtasks if I go to make new TKLDEV systems that use my buildtasks.  However, it seems easy enough just to change it manually as I have  just done to my working TKLDEV server.

  • 3 Changes made in Buildtasks (XEN) directory
  • Committed Changes
  • Pushed Changes
  • Went to Putty:  Git Pull  (3 Changes Made)

Lets Test a build

UPDATE:  10/23/2019

The Build process worked well.  I then had to wrangle (again) with network settings inside of my Server Build, but those were not related specifically to the TKLDEV process.

From here I will move the changes I made into my own BT-process and see if I can replicate my results and also get the few more tweaks for my server build integrated as well.

I need to understand a-bit about the Patching process and how it applies to files outside of build tasks  (ie, small changes to the Core build).  I also will need to test various updates from REPOS and TKLBAM to see that I can have my changes from TKLDEV stick.

All for Now.  Very excited about the project.

Print Email

TKLDEV Setup and Note taking

TKLDEV is the build system for TurnkeyLinux applications.  It is very capable and somewhat easy to work with.

I am wanting to build some more capabilities into TKLDEV and so today I have started again with the process of working with it and shold be able to proceed with modifying it.

To do so,  I installed from the TurnkeyLinux Hub, a TKLDEV Micro instance.  As the name holds, it is a pretty small build.  1 GB ram,  10gb of HD space.  Cost is not low:  About $12 per month, but it can be killed pretty easily and built again.

First Task:

Working through the lack of documentation with TurnkeyLinux.

This can be quite problematic.  However, if you read, and reread, forum entries, look at github documentation, and test and try things, things can work out.

There are several steps required for actually getting BuildTasks to start working.

  1. Get VMWare's OVF Tool working 
    1. Login to VmWare
      • IE: my.vmware.com
    2. Download the tool to your computer
    3. Login via FileZilla to your TKLDEV appliance
      1. Ideally you will have a "downloads" or similar folder for files
        upload your OVF Bundle File to that directory.
    4. In Shell, be sure the file has chroot 755 permissions 
      chmod 755 VMware-ovftool-(VERSION.and.control.name).bundle
    5. Run the install code on your TKLDEV server inside SSH  (use the correct syntax)
      ./VMware-ovftool-(VERSION.and.control.name).bundle
    6. Run the Program
    7. Go through multiple pages of "licensing code"  (Accept All)
      1. accept the Terms at the very bottom of the licensing agreement
    8. Then Create a Config Folder and Copy the Contents as outlined here:  
    9. https://github.com/turnkeylinux-apps/tkldev/blob/master/docs/advanced/optimized-builds.rst
      Configuration & Setup
      • cd buildtasks/
        mkdir config
        cp config.example/* config/.
      • run bin/vm-setup
  1. Interpret the correct Syntax inorder to get various BT-s working
    1. Hint:  actually type "./" where instructed to do so.
    2. ie: "./bt-iso"  or "./bt-vm"

Issues:
Verification of ISO's  (work through this)

Building from TurnKey ISOcd 

Note that if there isn't already a Core ISO and matching hash file in /mnt/isos buildtasks will download it from the TurnKey mirror.

When that's finished, you should find an official Core TurnKey ISO (and matching hash file) in /mnt/isos/ and a Core container build in /mnt/builds/container/

Turnkey Mirrors are sometimes downloaded rather than ISO simply generated  (see note 2 lines above)

Building from your own ISO

cd buildtasks/
./bt-iso new-app


Once that runs, simply run the relevant bt script. Don't forget to install the relevant dependencies.

./bt-vm new-app-14.2-jessie-amd64

ToDo's:

  • Get Git Working
  • Make Variations to build scripts
  • Practice, Practice Practice
  • Get Shortlist of "New App Setup"

 

Turnkey Mirror Links

 

Assorted Links of note:  

 

 

Issues at hand:  

  • Cloning
  • Making New Code
  • Using the new code
  • Integrating into the repository or simply using the repository as a base, but using local changes

Update Note 10-22-2019

  • I have gone through the OVFTOOLS notations above to be clear as to process.
    FKIT has made a good video here:  https://www.youtube.com/watch?v=vseorvzwu-k
  • I have started forking my own repositories and working on new builds.  Doing one for Joomla3 right now.  Forking TKLDEV and Buildtasks and bringing them in to see if they work with few changes.  First MAKE is running now.
  • Did also upgrade my Putty Application to 64BIT and current.
  • Waiting for the build as we speak.  Takes a while on a small server.
  • Once build it is important to look into the /mnt folder for names created, but also to understand the bt-vm command above for the first build.

Print Email

Building Joomla's Fields Component when it does not exist in your System

I do not know why this came up, but the Custom Fields component which was introduced to Joomla in version 3.7 did not exist on my system and it would not install on Joomla Updates.

Quite a bit of configuration patience went into getting it installed.  One of the issues really was the "NA" statement from the Joomla "Discover Panel".  It indicated that there was nothing to install, and Update indicated the same.

The trick was to extract the Full current Package Install into a separate folder (in my case a separate drive all together) then to move and merge the contents of the Components/com_fields folder.  

However additional configuration was required, again using the Discover and Install aspect of Joomla's  Manage Menu Set.  Not only did the component need to be installed but the System component (plugin I believe actually) had to be enabled, and the different Field elements also had to be installed and subsequently enabled.

Finally, Cache needed to be cleared and everything showed up.  Viola.  Happy me, though, I am not really sure I want to use that aspect of the system.  Knowing me though, I probably do.  I will post more here or nearby if there is anything of note.

Oh, and by the way, don't just extract the Joomla Package onto your working system.  I did that and lost system control.  I did do a full backup just before this so I was able to pull an immediately preceeding restore.  However, it was from that file set that I was able to pull the component/com_fields folder.  Worked like a charm.  I need to unmount that drive next.  Easy enough.

Print Email

Moving forward with Multi-Domain Setups

In Joomla there is a sequential approach that needs to happen when setting up a Multi-Domain environment.

Of course, the first part is running down the extensions you need, testing them, learning them, implementing them.

The second part is a continuation of the first.  More specifically though, you want to build out your Array of Resources.  I would prefer this to be a checkbox rather than a Bullit List but here we go.

First Requirements

  • Stable and Controlable, generally Linux based Server - lots to this but I can help.
  • Joomla Up to Date - should be easy
  • As many of your Extensions up to date - should be easy unless some subscriptions have expired - move on
  • Disable or Remove unused Extensions -  Many things you can do under the "Manage" tab on the backend of Joomla

Add Specifically

  • Virtual Domains
  • Smart Search (buildin)
  • OS Map (joomlashack)
  • Other Mappers for testing
    • Aimy SiteMap is helpful.  JSitemap is helpful, XML Sitemap Pro - external, is helpful
      • turn them off when you are not using the extras and don't submit to Search engines unless you need a second one for some reason.  Lots to discuss there.
  • OS Content (a great way to start your content buildout)
  • OS Meta (a great way to be sure you have covered your bases)

Remove:  

  • Search Modules

Add:

  • Smart Search Categories for each Menu
  • Smart Search Modules to each Menu
  • Custom Module that links to each of your Virtual Domains
  • Second Custom Module with Restricted Access for your in development Virtual Domains.
  • You can also do this with Articles but I find it too slow
  • SSL Certificate(s) for your domains (assumably applied in Apache)
  • Google Analytics Code to each Template being used by a Domain, and segregated to that Domain
  • Site Map registrations to google for each Domain  (you need to organize yourself here with Menus)

Test:

  • You will test alot, so keep your initial structures as simple as possible
  • Be Patient and Persistent.
  • You should do the same tests on each of your domains sequentially.
  • When all is working, move on to the next
  • Keep a good Checklist for Issues.  You can even write notes to your system with restricted access

Know:

  • You will get errors if you are repeating content in your menus the wrong way.
  • That means,  do not include the same content in two different Menu Items exactly the same way.  One needs to take precedence.
  • Move from one subject to the next throughly
  • Joomla ACL is very powerful.  It is complicated.  It is useful.  It is not always obvious what changes, so test the differences in settings.  
  • You generally won't kill your site, but sometimes you may need to remove something you just added that made your site stop working.
  • Big Changes should be preceeded with a System Backup that you can restore from and which you know how to restore from
  • I repeat:  Big Changes should be preceeded with a System Backup, manually triggered.
  • Have a Daily Backup system in place as well.

Check:

  • Your Virtual Domains Menu Filtering regularly.
    • If you add Menus Elements in Joomla you must add them to the "don't filter" (Menu Filter) set if you want to see them.
    • You can spend extended periods of time thinking your system is broken just because of this or some "bad key stroke sequence there.

Comment

    • An update to Virtual Domains was done recently, and it is clear that there can be more done.
    • All that said, it works very well and can save $ Time and Mind.  
    • Check out that Formatting Trick.  Never had 2 Bullets stacked before.
    • Have some fun and have some things to talk about other than your web site  (I am taking notes).

Print Email

Store Categories

Joomla3 Appliance - Powered by TurnKey Linux