Excerpt |
---|
If you found that your server shows "Proxy Error" - "Error reading remote server", it might be associated with the system resources or settings. |
I tried below method to resolve the problem.
Case Sensitive collation in your MySQL
If you are using MySQL as database, you should make sure if your database is created as below command. If not, you should backup your confluence, drop and create your database.
Code Block |
---|
CREATE DATABASE confluence CHARACTER SET utf8 COLLATE utf8_bin; |
I thought my database is created as utf8_bin, so may have no issue, but actually you will face lots of issues very soon.
Increase the value of maxThreads(default:48) on server.xml
The below is what I tried on server.xml - I just put 1024 for maxThreads instead of its default value(48)
Checking if Synchrony is working or not
I just checked listing ports like below, and I learned there is no port open for synchrony - its default port number is 8091
Code Block |
---|
~# netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp |
Code Block |
<Connector0 port="8090" connectionTimeout="20000" redirectPort="8443" 0 0.0.0.0:25 maxThreads="1024" minSpareThreads="10" 0.0.0.0:* LISTEN enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8" 2755/master tcp 0 0 127.0.0.1:6379 protocol="org.apache.coyote.http11.Http11NioProtocol"/> |
Apache proxy
If you are using Apache proxy, you should check if you change the setting like below. Note that you should put your own domain name for proxy server or just 127.0.0.1 is good enough if that is installed internally.
Code Block | ||||
---|---|---|---|---|
| ||||
<Server port="8000" shutdown="SHUTDOWN" debug="0"> <Service name="Tomcat-Standalone"> 0.0.0.0:* LISTEN 2104/redis-server 1 tcp 0 0 0.0.0.0:11211 <Connector port="8090" connectionTimeout="20000" redirectPort="8443" 0.0.0.0:* maxThreads="48" minSpareThreads="10" LISTEN 2086/memcached tcp 0 enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8" 0 0.0.0.0:22 protocol="org.apache.coyote.http11.Http11NioProtocol" 0.0.0.0:* LISTEN scheme="http" proxyName="127.0.0.1" proxyPort="80"/> 2088/sshd tcp6 0 <Engine name="Standalone" defaultHost="localhost" debug="0"> 0 :::25 <Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4"> :::* <Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true"> LISTEN 2755/master tcp6 0 0 <!-- Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties --> :::8090 :::* <Manager pathname=""/> LISTEN 29399/java tcp6 0 <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/> 0 :::443 :::* </Context> <Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" LISTEN 25560/httpd tcp6 0 0 127.0.0.1:8000 reloadable="false" useHttpOnly="true"> :::* LISTEN <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/> 29399/java tcp6 0 0 :::3306 </Context> </Host> :::* </Engine> </Service> </Server> |
Checking router settings if you forward Synchrony port
Based on the log on my router, I got lots of DDoS attack from external servers, and it does not necessary to open that port. Technically Confluence already has a proxy feature to support as below - the action you need is to delete that port(default:8091) in the forwarding list.
Reference: https://confluence.atlassian.com/doc/possible-confluence-and-synchrony-configurations-958779064.html
Modifying /etc/security/limits.conf in my CentOS 7 server.
Code Block | ||
---|---|---|
| ||
* hard nofile 500000
* soft nofile 500000
root hard nofile 500000
root soft nofile 500000 |
Increasing max memory by modifying setenv.sh
One of the possibility is caused by memory limits. In my case, mx used the default value (1024m), so I changed its size to 2048m.
Below is my configuration at <confluence home directory>/bin/setenv.sh
Code Block | ||||
---|---|---|---|---|
| ||||
# Set the JVM arguments used to start Confluence. For a description of the options, see
# http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
CATALINA_OPTS="-XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution ${CATALINA_OPTS}"
CATALINA_OPTS="-Xloggc:$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:G1ReservePercent=20 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.awt.headless=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.plugins.enable.wait=300 ${CATALINA_OPTS}"
# CATALINA_OPTS="-Xms1024m -Xmx1024m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Xms1024m -Xmx2048m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Dsynchrony.enable.xhr.fallback=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.locale.providers=COMPAT,SPI ${CATALINA_OPTS}"
CATALINA_OPTS="${START_CONFLUENCE_JAVA_OPTS} ${CATALINA_OPTS}"
CATALINA_OPTS="-Dconfluence.context.path=${CONFLUENCE_CONTEXT_PATH} ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing ${CATALINA_OPTS}"
|
Once your modification is done, you should restart your server to take effect.
Adding more memory to Synchrony
The default memory size for Synchrony is 1g, and below shows the memory is assigned by 2g. That should be added at the end of <confluence home directory>/bin/setenv.sh
Code Block | ||
---|---|---|
| ||
CATALINA_OPTS="-Dsynchrony.memory.max=2g ${CATALINA_OPTS}" |
Modifying a value of maxThreads.Connector in <confluence_home_directory>/conf/server.xml
Actually this approach is not recommended, because it could cause poor performance delay in GC(Garbage Collection) sometimes.
Nevertheless, if you still want, you may able to see maxThread as a member of <Connector>. The default value of maxThreads is 48 - I just modified it to 4096
Code Block | ||
---|---|---|
| ||
<Server port="8000" shutdown="SHUTDOWN" debug="0"> <Service name="Tomcat-Standalone"> <Connector port="8090" connectionTimeout="20000" redirectPort="8443" LISTEN 2651/mysqld tcp6 0 0 :::11211 :::* LISTEN 2086/memcached tcp6 0 0 :::80 :::* LISTEN 25560/httpd tcp6 0 0 :::22 :::* LISTEN 2088/sshd udp 0 0 127.0.0.1:323 0.0.0.0:* 1807/chronyd udp 0 0 0.0.0.0:11211 0.0.0.0:* 2086/memcached udp6 0 0 ::1:323 :::* 1807/chronyd udp6 0 0 :::11211 :::* maxThreads="4096" minSpareThreads="10" enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8" protocol="org.apache.coyote.http11.Http11NioProtocol" 2086/memcached |
If it is, go to "General Configuration" → "Collaborative editing" → Change "editing mode" → Save
Case Sensitive collation in your MySQL
If you are using MySQL as database, you should make sure if your database is created as below command. If not, you should backup your confluence, drop and create your database.
Code Block |
---|
CREATE DATABASE confluence CHARACTER SET utf8 COLLATE utf8_bin; |
I thought my database is created as utf8_bin, so may have no issue, but eventually faced lots of issues that associated with character set. Once I created a table by above commands, I had no issues for shortly.
Modifying a value of maxThreads.Connector in <confluence_home_directory>/conf/server.xml
The below is what I tried on server.xml - I just put 1024 for maxThreads instead of its default value(48)
Code Block | ||
---|---|---|
| ||
... scheme<Connector port="http8090" proxyNameconnectionTimeout="your_domain.com20000" proxyPortredirectPort="808443"/> <Engine name="Standalone" defaultHostmaxThreads="localhost1024" debugminSpareThreads="010"> <Host name="localhost enableLookups="false" acceptCount="10" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4"> <Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true"> <!-- Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties --> <Manager pathname=""/> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/> </Context> <Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" reloadable="false" useHttpOnly="true"> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/> </Context> </Host> </Engine> </Service> </Server>URIEncoding="UTF-8" protocol="org.apache.coyote.http11.Http11NioProtocol"/> ... |
Below is another example
Code Block | ||
---|---|---|
| ||
<Server port="8000" shutdown="SHUTDOWN" debug="0">
<Service name="Tomcat-Standalone">
<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
maxThreads="4096" minSpareThreads="10"
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
scheme="http" proxyName="your_domain.com" proxyPort="80"/>
<Engine name="Standalone" defaultHost="localhost" debug="0">
<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">
<Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
<!-- Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties -->
<Manager pathname=""/>
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>
<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0"
reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>
</Host>
</Engine>
</Service>
</Server> |
Once your modification is done, you should restart your server to take effect.
Apache proxy
If you are using Apache proxy, you should check if you change the setting like below. Note that you should put your own domain name for proxy server or just 127.0.0.1 is good enough if that is installed internally.
Code Block | ||||
---|---|---|---|---|
| ||||
<Server port="8000" shutdown="SHUTDOWN" debug="0">
<Service name="Tomcat-Standalone">
<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
maxThreads="48" minSpareThreads="10"
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
scheme="http" proxyName="127.0.0.1" proxyPort="80"/>
<Engine name="Standalone" defaultHost="localhost" debug="0">
<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">
<Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
<!-- Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties -->
<Manager pathname=""/>
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>
<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0"
reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>
</Host>
</Engine>
</Service>
</Server> |
Checking router settings if you forward Synchrony port
Based on the log on my router, I got lots of DDoS attack from external servers, and it does not necessary to open that port. Technically Confluence already has a proxy feature to support as below - the action you need is to delete that port(default:8091) in the forwarding list.
Reference: https://confluence.atlassian.com/doc/possible-confluence-and-synchrony-configurations-958779064.html
Modifying /etc/security/limits.conf in my CentOS 7 server.
Code Block | ||
---|---|---|
| ||
* hard nofile 500000
* soft nofile 500000
root hard nofile 500000
root soft nofile 500000 |
Increasing max memory by modifying setenv.sh
One of the possibility is caused by memory limits. In my case, mx used the default value (1024m), so I changed its size to 2048m.
Below is my configuration at <confluence home directory>/bin/setenv.sh
Code Block | ||||
---|---|---|---|---|
| ||||
# Set the JVM arguments used to start Confluence. For a description of the options, see
# http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
CATALINA_OPTS="-XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution ${CATALINA_OPTS}"
CATALINA_OPTS="-Xloggc:$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:G1ReservePercent=20 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.awt.headless=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.plugins.enable.wait=300 ${CATALINA_OPTS}"
# CATALINA_OPTS="-Xms1024m -Xmx1024m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Xms1024m -Xmx2048m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Dsynchrony.enable.xhr.fallback=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.locale.providers=COMPAT,SPI ${CATALINA_OPTS}"
CATALINA_OPTS="${START_CONFLUENCE_JAVA_OPTS} ${CATALINA_OPTS}"
CATALINA_OPTS="-Dconfluence.context.path=${CONFLUENCE_CONTEXT_PATH} ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing ${CATALINA_OPTS}"
|
Once your modification is done, you should restart your server to take effect.
Adding more memory to Synchrony
The default memory size for Synchrony is 1g, and below shows the memory is assigned by 2g. That should be added at the end of <confluence home directory>/bin/setenv.sh
Code Block | ||
---|---|---|
| ||
CATALINA_OPTS="-Dsynchrony.memory.max=2g ${CATALINA_OPTS}" |
{Once your modification is done, you should restart your server to take effect.
Change port number for collaborative editing
...