Appliance Admin Management

Overview

This guide provides system administrators additional information on the MangoApps technology stack and steps to perform basic system-administration-related tasks on the appliance.

The latter part of this guide assumes that you are logged into the appliance (have shell access). If you do not have shell access, please contact your MangoApps account manager or email [e-mail].

Outside of monitoring, starting, or restarting the MangoApps processes, please do not make any changes to the appliance software. Any changes to the software stack including any new software package or update you install is not covered under regular support and will be treated as a support incident. (chargeable at $200/hr.)

What are the different components of MangoApps and how do these components work together?

MangoApps appliance runs Ubuntu 14.04 LTS operating system and is made up of following components:

• Apache:Apache is configured to listen on port 80 and port 443. Apache, in addition to directly serving the static resources, runs an Apache module: Passenger (the Ruby on Rails container). There are other Apache modules that MangoApps needs. The complete list is available under /etc/apache2/mods-enabled directory.

  • Latest Version: Apache/2.4.x (Ubuntu 14.04)

• Passenger:Passenger is an Apache module for deploying Ruby web applications.

  • Version: 5.x or higher

• MangoApps Application Server: MangoApps application server is built on Ruby on Rails framework and is responsible for handling all user requests.

  • Version: Varies (more info below)

• Messaging and Document Server (a.k.a Janus): MangoApps messaging and documents server provides real-time messaging, and document related capabilities.

  • Version: Varies (more info below)

• MySQL:The MySQL database is the default data store. For a single node installation, it is configured to listen on localhost (127.0.0.1) on port 3306 and does not have direct external connections. The application server server and the messaging/document server access the database using ActiveRecord and JDBC respectively.

  • Version: 5.6.x

• Resque Workers, Scheduler and Redis Server: These components process background processing of various tasks including email

• Memcached:Memcached is used by both the frontend and the backend servers to store transient information for quick access. It runs on standard Memcached port

  • Version: 1.4.2 or higher

• MemcacheDB:MemcacheDB is used as fast, semi-permanent memory store to enable quick and reliable message routing by Janus

  • Version: 1.2.0 or higher

• Postfix:Postfix is used as an MTA to send system email generated by MangoApps

  • Version: 2.7.0 or higher

• Crontab based tasks:MangoApps configured for Cron tasks to perform certain tasks at regular intervals.

• Puppet Agent:(For auto-update and node configuration) Puppet agent connects to the MangoApps appliance server to receive application and configuration updates. The appliance VPN connection is required to connect to the MangoApps appliance server and receive these updates.

  • Version: 2.7.3

The following startup scripts to start, restart, or stop various components of MangoApps.

• Apache/Passenger (Required):

  • To start:/etc/init.d/apache2 start

  • To stop:/etc/init.d/apache2 stop

  • To restart:/etc/init.d/apache2 restart

• Messaging Server (Required):

  • To start:/var/www/mjanus/bin/janus_startup.sh start

  • To stop:/var/www/mjanus/bin/janus_startup.sh stop

• Document Viewer (Optional):

  • To start: /opt/mango/bin/fileviewerwatchdog.sh start

  • To stop: /opt/mango/bin/fileviewerwatchdog.sh stop

  • To restart: /opt/mango/bin/fileviewerwatchdog.sh restart

• Document Deep Search (Optional):

  • To start: /opt/mango/bin/deepsearchwatchdog.sh start

  • To stop: /opt/mango/bin/deepsearchwatchdog.sh stop

  • To restart: /opt/mango/bin/deepsearchwatchdog.sh restart

• Database (Required):

  • To start:/etc/init.d/mysql start

  • To stop:/etc/init.d/mysql stop

  • To restart:/etc/init.d/mysql restart

• Mail Server (Required):

  • To start:/etc/init.d/postfix start

  • To stop:/etc/init.d/postfix stop

  • To restart:/etc/init.d/postfix restart

• MemcacheDB (Required - Real-time messaging and presence may not be accurate without it):

  • To start:/etc/init.d/memcachedb start

  • To stop:/etc/init.d/memcachedb stop

  • To restart:/etc/init.d/memcachedb restart

• Memcached (Required):

  • To start: /etc/init.d/memcached start

  • To stop:/etc/init.d/memcached stop

  • To restart:/etc/init.d/memcached restart

• VPN (Required for initial install, Optional after that. VPN is required if you would like to receive application and configuration updates automatically)

  • To start:/etc/init.d/openvpn start

  • To stop:/etc/init.d/openvpn stop

  • To restart:/etc/init.d/openvpn restart

• Resque Workers, Scheduler and Redis Server:

  • Stop Workers as Rails User:su rails -c "/usr/local/bin/felixenv script/stop_workers.sh"

  • Stop Scheduler as Rails User:su rails -c "/usr/local/bin/felixenv script/stop_scheduler.sh"

  • Start Workers as Rails User:su rails -c "/usr/local/bin/felixenv script/start_workers.sh"

  • Start Scheduler as Rails User:su rails -c "/usr/local/bin/felixenv script/start_scheduler.sh"

  • Start Redis Server:service redis-server start

• Deliver Emails:

  • To start:/opt/mango/bin/deliveremails.sh start

  • To stop:/opt/mango/bin/deliveremails.sh stop

  • To restart:/opt/mango/bin/deliveremails.sh restart

What monitoring tools are available to monitor a MangoApps instance?

MangoApps comes with a set of tools to enable easy server and applications monitoring. At a high level, we divide monitoring in two parts

1. In node monitoring:

   • System monitoring: A set of Unix scripts are used to monitor node level metrics like memory usage, disk usage, and ManogApps Application processeses and take corrective action. Run “crontab -l” to see a list of all the configured scripts on any given node. All scripts are typically located under /opt/mango/bin. In addition to the scripts listed below, for private cloud customers, MangoApps uses AWS Cloudwatch to monitor system level metrics like CPU, network activity etc. On-premise customers are encouraged to use the various tools at their disposal (VM monitoring tools, SNMP agents etc.) for things like CPU, Network bandwidth etc.

     • Disk and Log file check: Runs every 5 minutes to checks for available disk space and if needed, deleted old log files and sends an alert email. By default, the email goes to a mangoapps email address. Please provide an email address if you want this email to go to someone withing your organization.

       In addition, to manage disk space, logs are configured to be rotated by logrotate (a ubuntu utility) on a nightly basis.

     • App Server Memory watchdog: Runs every 3 minute to check for memory used by any single app server process and restarts the process if memory exceeds configured limit.

     • FileViewer watchdog: Runs every 27 minutes to make sure the file viewer is running. If not, it restarts it.

     • Deep Search watchdog: Runs every 10 minutes to make sure the deep search watchdog is running. If not, it restarts it.

     • Deliver mail watchdog: Runs every 10 minutes to make sure the mail delivery process is running. If not, it restarts it.

     Other, higher-level, monitoring are,

     • Database monitoring: MySQL ships with a database watchdog that restarts the DB if it ever goes down

     • Application Server: As this impacts the end-user right-away, external system monitoring ensures overall service is up and running and alerts in-case of a failure.

   • Performance Monitoring: MangoApps systems can be configured to work with NewRelic for extensive performance monitoring. By default this is disabled for on-premise setups. If you decide you want to enable performance monitoring, please provide us with a newrelic account id and we will configure it for you

2. External Monitoring: MangoApps uses pingdom to provide external monitoring of its cloud systems. For cloud customers, Pingdom is be configured to send a request to MangoApps every minute from servers around the world. These requests from Pingdom validate the MangoApps service from an end-user perspective. On-premise customers can signup for pingdom or a similar service to externally monitor their setup.

   • Application server monitoring: Configure the monitoring system to send a request to your MangoApps login page ( http://<your instance’s mangoapps fqdn>/u) and make sure the response code is success

   • Messaging server monitoring: Configure the monitoring system to send a request to http://<your instance’s mangoapps fqdn>/janusstatus and ensure that response is OK.

Can I install my own infrastructure monitoring and SNMP systems to send alarms?

Yes! Some of our customers are using Nagios and others use other SNMP agents. As monitoring tools and requirements vary widely from customer to customer, MangoApps does not install these agents for you. However, you can do that and setup these agents exactly how you like them.

Where are the key log files located?

• App Server:

  • /var/www/sg/log/apache-access.log

  • /var/www/sg/log/apache-error.log

  • /var/www/sg/log/production.log

  • /var/www/sg/log/mailer.log

  • /var/log/apache2/error.log

• Messaging server:

  • /var/www/mjanus/log/janus_server.log

• Database server:

  • /var/log/mysql/error.log

• Mail Server:

  • /var/log/mail.log

  • /var/log/mail.info

  • /var/log/mail.err

  • /var/log/mail.warn

• VPN:/var/log/syslog

• Background and Other utility Services:

  • /var/log/memcachedb21201.log

  • /var/log/memcachedb21202.log

  • /var/log/memcached.log

  • /var/www/sg/log/resque_scheduler.log

  • /var/www/sg/log/resque_workers.log

How can I verify all the services are running?

If you have a single node installation, following services must be running on that node. If you have a multi-node setup, depending on the role a node plays, these services are spread across your nodes.

• Application Server

  • Apache / Passenger

• Database Server

  • MySQL

• Mail Server

  • Postfix

• Messaging Server

  • Java / Janus

• Document Server

  • Java / Janus

  • Solr (Deep File Search)

• Background / Cache & Utility Services

  • resque_schedule / resque_workers

  • deliver email

  • cron tasks under user root and rails

  • Memcached

  • MemcacheDB

• VPN

You can see the status of various services using pstree -ap orps –efor by executingnetstat -tlpnu. e.g.

• To check if a process is listening on the correct IP and Port

  • netstat -tlpu | grep <process name>

• ​To check what all processes are running on a particular node

  • ​pstree -ap | more

How is the database backed up? How often? Where are the backups located?

By default, the database is backed up on a daily basis by a Cron job using the program “mysqldump”. A full dump of the database is made so that restoration is easy and fast.

All database backups are located under/opt/mango/db/archive. These database dump files are stored in a gunzip format and follow a file naming convention that makes it easy to locate dump for a specific day (e.g.mangoapps-127.0.0.1-20120126-003001.mangodb).

By default, the data backups are retained for 10 days. The script to back up the database is located under/opt/mango/bin/mangodb.shand is configured to be called nightly from a Cron task. Please make sure this process is working as part of your on-premise setup process.

A notification email can be configured to receive status of the nightly backup. By default, this email address is set to [e-mail]. This should be changed to whatever email address you would like these notifications to go to.

How do I restore the Database from a backup file?

Please see this article for details: https://guides.mangoapps.com/on-premise-and-private-cloud-guide/on-premise/how-do-i-restore-backed-up-data-and-files-to-my-mangoapps-on-premise-domain

Please make sure that you really want to overwrite your current database before restoring from a backup as this operation is irreversible.

How do I backup documents uploaded to MangoApps?

For our cloud customers, MangoApps automatically backup the files on a nightly basis. For our on-premise customers, they can easily backup the location all the files are configured to upload (location is configured from the mangoapps admin interface) using backup tools and processes they have in place. If you need help in setting up rsync or similar utility to backup files to a secondary location, contact your customer success manager and we will are happy to help.

What about Disaster Recovery?

Outside of the database and the document storage system, MangoApps is a stateless service (all the other states get recreated from configuration and installation files). So backup/restoration primarily applies to the database and the file storage system. As long as database and file storage system is getting backed up properly, MangoApps service can be fully restored from configuration and installation files managed by MangoApps Configuration & Update Management server. For restoring the database, please see the database restore section above. For file restore, please refer the document backup section.

How do I know what version of MangoApps the appliance is running?

To see the current version of the application you are running, please login to MangoApps and from the Help icon in the top right corner, go to About menu item. Application Server and Messaging/Document server version numbers are located in the about dialogue box.

Are there commands I can use to stop and restart all Mango services?

No one command is available to restart all of MangoApps. You can use the start / stop commands above to start individual services or you can gracefully reboot the machine(no hard reboot please!). All services are set to auto-start when the server is rebooted.

Do all the services auto-start on reboot?

Yes! All services are set to auto-start on reboot. If some services are not starting, please check the appropriate log files and use the appropriate command above to start them.

What ports are required by MangoApps and for what purpose?

The following ports must be open for inbound traffic from (client Initiated “Inbound Connections):

• HTTPS: 443 (Used for all MangoApps communication experience)

• XMPP: 5222/5223 (The client opens and maintains a persistent socket connection to push IM, presence and updates in real-time)

• Flash: Port 843 (Flash player socket policy)

• RTMP: 1935

• SMTP: Port 25

The following server-initiated “Outbound” (to Internet) connections are made by MangoApps:

• Software Updates / Downloads: Port 80, Port 443

• SMTP : Port 25

• TCP 2195 to gateway.push.apple.com (iOS push messages)

• https://android.apis.google.com/c2dm/send (for Android push)

• Active Directory port: Configurable from Admin UI

• IMAP and POP3 port: Configurable from Admin UI

• For multi-host setup, we recommend that all internal traffic is allowed on all ports

What Application-specific Cron / periodic tasks / background tasks are running?

• MangoApps application specific tasks : 15 minutes, hourly and daily

• Database backup: Once per day

• Email Notifications: Always runningdeliveremails(background)– responsible for sending email notifications.

How do I configure the mail server on my appliance?

MangoApps uses PostFix as its MTA. The configuration files are located in/etc/postfix/directory. The file you would want to modify is/etc/postfix/main.cf. As mail servers can be configured in many different ways, please consult http://www.postfix.org for detailed configuration instructions. In general, it is sufficient to just change the mail relay server IP to the mail server you would like to relay all the emails from. Don’t forget to restart postfix for configuration changes to take effect (/etc/init.d/postfix restart).

How often MangoApps does Major software release?

At MangoApps, we practice agile development methodologies and average one major and many minor releases of MangoApps every quarter. Each release contains new features, improvements to existing features and regular bug fixes. Critical issues during a major release cycle, if any, are deployed using patch mechanism built into our deployment process on a priority basis. Please see https://www.mangoapps.com/releases/ for additional details.

Everyone is seeing an HTTP 500 error page. What should I do?

Follow these steps…

• Please look at the HTTP 500 error page and see if it lists the problem.

  • If it is a log file permission-related issue, delete the log file (It will get regenerated).

  • If MangoApps could not connect to the database, see if MySQL is running. If not, please start it.

• Check if all necessary services are running.

  • If something is not running, start it from the command line.

  • If the service is not starting, please check the logs for that service and see if that indicates a problem.

    • If it is a log file permission-related issue, delete the log file. (It will get regenerated.)

    • If the issue isn’t clear, give us a call.

• Try restarting all services.

• Try rebooting the appliance

• If all else fails, give us a call or send us an email at [e-mail].

Images and files are not getting uploaded. Where should I look?

Go to the web admin interface and make sure that that your storage settings are correct. Inspect the Janus log for any exceptions and error messages.

Instant messaging is not working. Where should I look?

Make sure Messaging Server is running. Also, make sure that required ports (5222 and 5223) are open.

Profile thumbnails aren’t showing. What should I do?

Make sure Messaging Server is running.

Some of my users are not able to connect over IM. What should I do?

Please check your firewall settings and make sure that port 5222 & 5223 is open for all users. Please see the Janus log for any exceptions or error messages.

My users are not receiving emails, where should I look?

Look in the mail logs and see if the relay server is configured properly.